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■ABSTRACT 



The goat of this project is to "find ways,. of enhancing'? > . 
the efficiency of searching large machine-readable data bases.' 
This -includes improving the recall ahd^pr^ecision characteristics 
of retrievals initiated by user, requests -as well as l\elping the 
user to^ form concepts. For the . iattep, ways are to be sought 
. to transfer to the computer _ some of the tasks tlCat- are normally 
performed by^t^ie user, i.e. to further automate iV (information ' 
retrieval); Such^ developments are motivated by thg" rapid growth 
in the volume of on-line IR' activities, ^d the fact^that the 
cost .'of searches is ho longer limited by cpu search, costs. ' .* 
Rather, it is limited by labor costs v (profiling, evalu'ating 
output., bookkeeping, etc.)' and I AO" costs-, (printing, mailing ^ etc.) 
Fot a typical search, costing between 100 and 300 dollars, usually 
less than $5.00 in cpu is consume4. Such cosf« siiggest ^hat '. 
large efficiency gains can be made by further automating IR 
systems, functions. Underlying these goals are two general issues. 
The first -is the relationship between statistical" string pro- 
cessing, and semantic word processing. The second is the concept 
o.f multi-step processing of a. search request . 

Statistical string processing pertains to tho's^'lR functions 
that can be performed without knowing the definitions of the 
terms (character strings), i.e. .sorting terms and grouping 
records on the basis' of the terms they contain. This is the 
typical, method used in Boolean searched and simple terta 
clusterin^g. ^. • ■ . 

• Semantic word processing pertains^ to those word relation- 
ships that depend on term definitions, i.e. the meaning of the 
term^in the conteJ5[t of the data. base.-, Multi-step processing- 
•df large files involves using more than one methodology in 
distinct steps, to process. a single search request. The steps 
are arranged so that the first process i.s most appropriate for * 
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use on a very large^file. The second 'step then operates on 
a subfile identified by the first "step ' and further refines 
the output file, etc. In this. 'sftudy, , tha multi-step search 
idea wa's tested at length, using Boolean search, as the. first ' 
step and subject ^term *clu§tering as the second. The results 
were encouraging^ Moreover, it was found that the processing 
^may be further strengthened by incorporating some semantic 
informat^dn into statistica!^ sCting processing, by the use of 
a new method of Automatic Term Classification (ATC)'. The 
ATC method , allows the string comparison, m'echanism Co either 
match the categories rather than match- the strings, op;^o 
,limit the compares to those terms that, lie within .4 giA^e'n 
category, ' The latuer process is new, and toorrespo'nd^l't^. the^ 
psychological process of focusing, attention on a limi^ied - .-.j. - • 
.family^pf record aspects. ' pv^erall, the results suggest an 
organization for the €>R system of' the future in which seT^efal- 
processing techniques are -useg^^ing adsingle retrieval,.. anjd 
in which the system will be a^ active search partner perforra^^. 
ing like an ideal libratiari. ' » .1 
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/*The rn>nd requires, a representation of knowledge wherein inteYas?ociated 
Jde^^ard Jabeled according -to thejir type. Such labeling seem^Tutterly 

necessary in order to ditect efycCent searclies through memory/for. 

•information that n^ets certain r^qui rementsT . ♦ • • . 

■ . '- ENHANCING THE ^RETRIEVAL EFFECTIVENESS' . 

. n ^ OF .IJOlGE^NFORiylATION^' SYSTEMS' ^' 

a. - BACKGRIQUN D . ' . . a' 

'^.During the past 20 years,/ the application of computer * ' 
technologj? to solving infbnna,tion retrieval * (IR) probleips has 

^^becpme- co'nnnonplacfe* These applications -are motivated by many 
^factors, .the mpst^ prominent of ^vhich are probably^ the advances 
in electronic data process in|5 and' Computer ..^fint setting 4- ,^ 

^ technblo^y, Che rnfboiacion explosioh^n^i fche r'ecognitiph by ' ' 
agencies,' primarily the National Science Foundatrion (NSF) , • 
t^atL.the '.cost-benefit rdtiop' favoring! rese'arch on\,IR technoiogy' 
aire enormous. • . 



To date, t'He Commercially viab.le..IIf- syst'eins for large 
, bibla^graphic'dat-a bases have not been "thi-nking" systems, ; ' 
in thfe sense that they identify records for ^ retrieval based ' 
on the; character;strings t'hat they, coiitain. - independent of the 
>confceptual .definitions of- those stringfe. 'For instance, one " 
may query ■Bod'>ean systems/f or co-occurrences (occurrehdes 
within one record) of th^p Itrings "ozone" .a^d "tdmato". -in order' • • 
to xdenti-fy those records pertfinent to* tne concept, "the effect 
of ozone on the tomato -plant. " In performing, thts 's^arch^ the ■ 
system dpes 'not make use" of the definition, of ozone as -a ' ' ^ ' 
molecule composed of ' three oxygen atoms nor- does it 4ibe the 
jiefiTi^ition'o^- tomato. Rather, the system merely sear;ches f'o^ 
.^ccuirrenc^ea 'ot- the explicit character strings i "ozone" and • 
.•"tomato". The systems of most organizations, work this .way' 
including.* ehe IIT Research Institute' s -Oomputer -Search Center ,TiSC), 
The Nati6nal Library, of Medicine. (Nlil), Lodkheed. Information ' 
Systems »^ SDC Search Service and the University of'Geprgia 
Information Dissemination Center. One exception is the -institute 



of Scientific Information „(ISI) sys^'em, which identifies ' - 
'.related records via rjeferehces cited' witKin" each docipiient.^ 
That is, the ISI system .g.ff^ctively sidesteps the .problems of. 
handling and manipulating sub j ect terms by stinking each ' ' 
record to -those* records that it^cites, jl^.^he future', it - , 
would b"e desirable to bo-ordinate t,his ' (Capability/ wliich is 
a natural exrtensiop. of manual, procedute's, with the suttject 
term oriented 'capabilities .studied ip ;this report-.* 

The^ enormous success of'-IR systems based .onrmer.ely - 
matchifhg character 'strltngs motivates one to try to automate * 
more of th^ steps in^ the process , corjceptifally outlined ^.off , 
^TabldU. The task 6f ^omp'osing a Combination of character' 
' string^ that ^will represent ^a giAfen -concept (prof ilingV-anci-^ 

retrieve^ apprjopriate records with good peifformance .is IdiVficult. 
^It requires knowledge of the sjiatistics of the terins ;vithi'n the 
data base as. well ^s' knowletige .abpiit the desirecf concept.. .. ' - 
Accordingly, -the- profiling t^sk is usually i^erformed by « 
information specialists.. Search failures can occur", for many 
reasons., including; failure to translate Ihe concept i-fltp the ' 
specific terminp;g)gy of the sys,tem, ^f^ilure to identify close.ly " 
related concepts 'and fajllure to learn during the cpuKse of th:^ 
search, tho.ye new concepts th^t ar.e related tg ^he old one' by 
implications - rather .tha^i over lap/ofi char%t^r strings. . ' 

Clearl>. some of the capabilities .'that one .would lii^e to^ 
automate 'in an iR system ate those of ^^^r^gal -librarian: the' ' 
■ability to sunmiarize the_gener^l„cndracteristics of a retrieyal^- 
or a colleMon without .nec,2s;s^fil^> having." to analyze all* the 
implications of the text in the records, the ability to'dis- . 
ambigtiate different clisses of term cp'-ocjurrences (i.e. 
distinguis'h between/'tlie effect- of pW on'tomatp plants" and " 
"the generation of ozone by tomato plants"), th^ ability to . \ 
suggest to the user certain aspects of the searcH that -are ' ' 
likely to be .of interest, etc. Becaifee these capabilities 
involve using terms as more than just charact^ strings, ,thyoy ' ' 
imply that the system will have to have available"' to it, some * 
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The User 



]. Conceptualizes document 
characteristics 



Expresses chara'cteFrsTics^ 
in -terms of Data Bas^ .and 
fR system 



3. Operates system and 
receives output 



^Evaluates output and~ 



^a. Is satisfied; .or 



4b- Modifies expression, or 



kc. Modifies concept, or 



kd. Terminates unsatisfied- 



Identiries l<nown authors^ corporate 
authors, subject areas, relatefl 
concepts, time periods*,, . ^ 

ldenTrfiies~Rey Wrds "and** subject 
index terms wi^th the subject areas, 
Jdeptifies relevant CA s'ection , 
^numbers. Adjusts time period for 
publ icatiori lag* • . . 

Refers to CAS Subject Index', Formula 
Index, Subject Guide and Author' 
Index for abstract humbers- Proceed 

to abstracts for references* • • 

" k. 
^ ♦ / 

Reads parts or all of abstracts and 
^^.^».mak6^s decisions as to completenjsss 
and relevance/.- 

Decides that search has exhausted 
CAS capabi 1 it ies and/or has fulfilled 
' search needs. 

includes related terjns, corrects^ 
errors of translation, .returns tfo^/ 
Step 3. ^ " ■ 

Corrects errors of thought or incor^ 
porates new ideas learned from search. 
Returns to\ step 2. 

Is frustrated,- runs out of tjme or 
money. ., . 



Same 



Same plus association of keywords 
and keyword fragments in logic 
statements, examination of keyword 
and fragment frequencies... 



Key input and operate computer 
system. Output computer printed 
citation cards, sometimes obtain 
full abstracts for references 

Same 



Same 



Same 



Same 



.Same 



TABLE. 1, Steps in Information Retrieval 



degree of conceptual term definition. Language processing 
using conceptual term representation is usually called , 
semantic information processing. ' 

Curiously, it has been found that attempts to incorporate 
semantj^c information into an information retrieval search 
mecHanism have generally resulted in degradation o.f search 
retrieval performance for, equal search cost, as compared 
with statistical string processing.^*"* That is, for a given 
dollar cost, a statistical string based search mechanism will 
generally give better performance than a system using semantic, 
information." 

Many of the attempts to inqprporate a degree of semantic 



information into IR systems have been reviewed by Montgomery^, 
'and" more recently by Damerau^ . The general structure of these 
^_^ys.tCTis-is shown in Figure 1,- adapted from. Montgomery^ . ^ 
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Formal ^ 
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of Data Bas^ . 
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Representation 
o'f" User Queries 



Records -E^ar 
Which Ma£ch 
Occurs 



'Matching \ 
Algorithm/ 



Tigure 1, IR Systems Design Based bn* Canonical Representation^ 



User queries and data base records are each translated into 
a formal representation that facilitates the recognition of ,^ 
matches between them. The choices for the format representation 



vary widely, including contribution-s-^from semantics and syntax 
of the data contents. , Some systems, such as those of Sager'' and 
Kuno-Oettinger® use a syntax-driven phrase structure grammar to 
•identify and rewrite recdrds into canonical forms. These systems 
are top-down in the sense that they used fixed rules to classify 
input strings. Transformational grammars, have also been applied. 
Other systems use semantics -driven procedure's to replace records 
with a xepreseritaCion in Semantic .primitives. The systems of 
Wilks^' and Laffal'^ are , of this type. Yet other systems com- 
bine syntactic and semantic information to approach a^ more 
complete representation of the data base. Von Glaserf ield ' s ^ ^ 
system is of this_ type. Finally, there are more comprehensive 
Artificial Intelligeupe (AI) systems, like those of -Svimmons^ ^ 
Schank^ '' and Winograd^^ which use "internal representations cliat 
approach the power of handling text in a cognitively meaningful 
manner. Such systems, ' of course, are m^ch more expensive to 
operate because of their high requirements for computer memory 
space and processing time. However, their capabilities a^e.^ 
impressive.' AI Systems exist" today that can input up to about 
one ^hort paragraph of English text, in a very limited context 
of discourse, can process it into an internal representatibn 
and then can^answer questions about it, phrased in nearly free 
English. The -existence of such systems today motivates "tht 
question ox what their relationship will be to the ik' system ' 
of the future. That is, are the statistical string 'techniques ■ 
that are dominant today at commercial s'ear.ch services destined 
to be r.eplaced by semantic techniques in the future, or is a 
sharing of roles more vLikely?. \ ^ ■ ' ' } 

i ' 

Because statisticall processes have been most ,i:ost-ef^icient , 
research has recently been done on ^enhancing the feiciency of 
these processes. A logical extension of the Bool^^an search 
procedure is to relate^ the probability of conceptual similarity 
between two records to the number o'f character sitrings that they 
hold in common. That is, recdrds containing, the s.ame strings 
are more likely to concern the same concepts than are records that 
don't. Using this principle, it is possible td partition record 
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collections into groups, or clusters, such that members within 
a group share vocabulary overlap, and, probably, concepts. 
Unfortunately, the cost of clustering ii\creases, rapidly as the 
►• file size increases, because it invoLves comparisons among all 
records. For a collection of Np* records, most clustering ' 
al'gorithms consume an amount of computer processing time pro- 
portional to between N^'lnCN^) and N?.^* 

For Instance,' if a file-with 100 records is. cltrstered using 
10 cpu, then a file of 10,000 records would require between 400.. 
cpu and 10^ cpu. Since many bibliographic files are much ' 
larger than 10,000 records, it is difficult to see how a clustering 
algorithm could be efficiently used on a large 'file during a - • 

single, on-line accession.. " • . < 

• ,< / " ■ . . 

' Using clustering on jsmall sets-, many investigators, prin- 
cipally G. Salton^^ and K. Sparck-Jones^ ' , have studied new 
designs- for JR- systems. ,,i^Salton generally .uses about 1,'doO 
records, and K. Sparck-Jon.es uses fewer. Via thi^ method, 
i-ecords are clustered into groups before retrievals are, done. 
Then, a user query may retrieve any of the already clustered ' 
record groups. This process is analogous tp retrieving all 
entries- under a subject category such as a Library of Congress . 
Catalog number. However, with clustering, the records may be 
conveniently raffked according to. theiJ probable relevance to the • 
search query. One feature of these systems thae has recently k "" 
beqn.^eicploited is that user judgements on relevancy of output may , 
be readily incorporated, -by automatic means, back into the 
retrieval rae.chanism so as to« re-prioritize the output. ^^'^^ ^hat 
•is; if a given record is rated as relevant, the terms in that 
•record can be more highly associated .with relevance and the terms 
not appearing can be m6re highly associated .with non-relevance". 
.The opposite procedure is applied for records judged to be non- 
relevant. The results of .these judgements are then applied to 
all candidate records, through th^erms they contain. Such- 
procedures are capable of very high IR performance in situations 
where many relevance judgements may be accumulated. In contrast, 
*A11 symbols used are defined in Appendix A. 
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it seems that for the case of on-line interactive retrieval, 
it wbuld be more efficient to have the searcher make the 
judgements directly on the terms themselves. Then, the system 
does not need a procedure to automatically weight the terms. 
Instead, it is told that information directly*. The key points 
developed by these workers that are relevant to the work to 
be discussed herein are: ^ 

1. Statistical methods exist for automatic partitioning 
of records into classes based on their tjerm oX^erlap; 
Clustering can eithei; be u^er independent or user 
dependent ; and ^ ,.j 

'Subject term clustering i^ ijjsually limited in 
application to small file's for reasons of processing 
cost. - 

liser relevance judgements made on one group, of 
records can be automatically e_x,trap9lated to another 
grovp' of records on the basis of their shared terms.' 



2. 
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, 4rogram concept - " ^ . 

. T\he central idea of this program is that more «than one 
search .methodology can be used during the* course of a single 
retrieval. Perhaps it isj the case' that IR systems incorpora- 
ting som^ degree of seman'^ic information processing . are less 
successful than purely statistical string processing programs 
because the statistical processing is the mpst* efficient single 
way to conduct a retrieval. That is , '.perhaps the various 
retrieval methodologies , cab be thought of as screens of varying 
coarseness, with Boolean s'tririg matching being 'Nearly ,the 
most'erudej clustering, for example, being less crude (because 
it uses all of a record's therms, rather than* only ^he .selected 
ones as occi;^rs for Boolean " ' " 
processing beipg much finer 



search) and semantic information 
If the. screen analogy i^.yalid, 
then the most ,bo?t-fff ective w^y to perform a very prec"]Lse 
searcji- is not' to apply ythe 'finest screen to every record. 
Ra-ther, it is/ to start with a coarse screen, and to' use it to 
separate out/all those items that, at its level of.coarseness , 
do 'not. apply/, and then to 'apply the more fine screens to the 



remaining ii^ems. This implies that the many forms *of canonical 
representadLort previous alliidea'to, and .their ^corresponding 
match me:chMsms, a^e all candidates for use4n ^co-operatit^e 
systems mor^ complex, than that shown iA Figure 1. That' is. Any 
combination of those systems could be arranged in a sequence of' 
steps. to process a jingle user query. Many combinations are 
attractive. For t}iis study. Boolean searching was chose'n as' 
the first step of an information retrieval . and siibject term 
clustering of the resultant ,set\of (Boolean, search selected) - 
records was chosen as the second step. ' • 

There are several factors. tAat motivate the icoupling of a 
Boolean. first step with d clustering second, step.! First ' 
Boolean techniques work: well, with inverted term files, so 'that 
they easily accomodate large files. Subject term clustering 
techniques, however, are prohibitively expensive for large files 
Second, whereas Boolean techniques require user specified terms 
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cluster techniques work on the contents of records, and so can 
accomodate the many highly specific low frequency terms that 
are so inaccessible to Boolean methods in pi^oducing the pattern. 
Also, because clustering operates' on the record contents, and, 
in effect, -summarizes the retrieval a's a,' pattern ,, the pattern 
can Assist, user concept formation about the tferm co-ordinations" 
that are represented in the retrieval. That is, IR is essen- 
tially a closed problem because the user can always sidestep the 
IR sys^tem and manually screen all the records, for^ the desired 
properties. Hence, the measure of the eff ecti-v^ness of any IR 
system is the degree to which it reduces the ntimber of user •' 
judgements-while preserving sufficient recall. By grouping 
Boolean-retrieved records, clustering c^h reduce the number^ of . us( 
decisions required to the number of "clustered groups. That 
is, if ali-recdrds in a group are similar ," then. only one or two 
of them need to be examined so. as to evaluate the relevance of 
all the members of the group. Second, the grouping provides a 
mechanism for feeding .back to the user summary level i-nformation 
about the characteristics of his retrieval set. For such a 
mechanism to be useful' it should perform at a cost less: than ... ' 

would be required for manual. feValuat ion of the 
retrieved setter other available means. 

Some might argue that it would be mo^e appropriate to couple 
a Boolean first step with a syntax based second step. It was ' 
decided to use clustering because contenp information, which is 
accessible to clustering methods, seems to be a more coarse screen 
than syntax information. After all, titles are an effective 
retrieval field, and-titles are usually ^)hrases , not sentences. 

It seems natural to first consider the terms that are present , • 

then their context, and. then their syntax. ' . ' 



THE RETRIEVAL PERFOia^CE £>'ROBLEM - TYPICAL' PARAMETERS 



/ 

TJie retrieval perfoinnance prpblem involves, the difficulty 
. one has in achieving high recall with high precision in, for 
instance, on-line bibliographic retrievals. This problem is 
illustrated in Figure ;2 for typical search parametters for an ' 
^on-line retrieval frqm a large data base.. If terms of very 
high •specificity are' used in the Boolean' retrieyal search, 
strategy (i.e. low frequency terms such ad the names of specific 
plants (pine, carrot, etc".)), the riuniber of records that satisfy 
t^^ search strategy (the retrieval set) is "sriatl, the precision, 
is high (most ra^rieved records are relevant) " but many relevant 
• records exB not/retrieved, becaucie they did not cdntairi- the 

- specific terms/ chosen by the, searcher. .Ifc, altrerna.tiveiy , 

- terms of lo^>r specificity are used in the search 'strategy (i.e. * 
' high frequency terms such as plants, botany, etc .7), .the number 

' .o£ records ^hat satisfy, the search strategy is "large, the 

precision i's low (many^ retrieved records are not relevant), but 
most relevant ^ecords^ are retrieved., Thus, there is a tradeoff 
■ .between the number of relevant records missed' and the uuer time 
required tp evaluatr? possible non-relevant' records . For -dif- 
ferent users, the tradeoff is usually satisfied by varying the 
size /of the retrieval- set. In Figure' 2; a retrieve'. l" of about " 
100 records results in a precision 'of about 30%. so that 30 
r^ievant and 70 non-relevant records are retrieved. A>more /. 
complete search, yielding a retrieval of 1,000 records results 
■in..a precis^ion of • about 10%, so that about 100 relevant records 

/ and 900. nofirrelevant records are retrieved. \ 

'Not all searches need be exhaustive, so not all users will- 
opt for the larger, more complete searches. At IITRl's CSC, 
however, exhaustive searches are of. ten. required, and so the'fol- 
, -lowing question arose. Suppose that the Boolean search parameters 
were a^anged to yield an exhaustive retr.eyal? I-s there any 
additional computer pr.ocessing that could be performed /on the 
retrieval set so as to further sep.arate the relevant from the 
non-relevant records? That is, the Boolean search technique, even 
when used with general ^terms so as to yield high recall, is still 
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Figure 2: Typical Recall - Precision Tradeoff as- a Function of 
Retrieval Set Siae for Booleaa Search Strategies. 



^ery effective filter, reducing the set of candidate records - 
r retrieval from perhaps 2,000,000 to perhaps 2,000, as 
lustratTed in Figure 3*. Now, the 2,000 item retrieval could/ * 
further- refined by additional Boolean restrictions. The 
roblem is that the formulation of those additional restric- 
ion.s would be very • time-consuming because they would- necessarily 
nvolve low frequency terms, and hence, a long and complicated 
s'earch strategy! Also, in or*der to formulate this long. and. 
refined search strategy, it is necessary to find out some of 
the s^ary level characteristics' of the retrieved set, arid 
the onYy way to Ho that_^now is to scan som^ of" those records or 
try to guess the terms that are present 'and to enter them as 
seatdh terms. However, why should a user have to guess? Wouldn't 
it be" better for. the computer to sort the characteristics of the 
relatively small retrieval set and report, them back'to the user? 
^he manual scanning'^r excess .of refining'^he' Boolean logic is sq 
slow that a user\ is .of teji bett.er off, when he requires^ an ex-' 
^haustive search, to simply print the entire -high recall set aJd 
manually reject the non-relevant items. If the retrieval set- 
C£ 2,000 records were'' partitioned into 20 clusters (of* 100 • 
records each) , and if all of the -relevant records were to be " 
in one cluster^, then identification, of that cluster' would yield 
a high recall search with high precision. The Boolean step 
would be^'ecall-oriented .and the clustering step v/ould be 
precision-oriented. The selection of the appropriate (high ' ' 
recall .with high precision) cluster^could then be accomplished " 
by, perhaps, examining one or' two sample records from edch ' 
clustery reducing the number of relevancy decisions from 2i000* 
to about 20. . - ^ ' . . • 
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Z. •> METHODS AND ^TERIALS ' .1 

DATA' BASES ' . ' 

- c ' ^ • • 

^ • The d^ta bas^ used forvche experiments, were -Chemical, 

A*DS tracts 'Services CACon-, VoliiAes 82 and 83 and Engitieering 

Index's COMPENDEX (Ei) , Volumes 74 and 75. CACon addr>2sees 

wide range of chemistry rel-ated literature'. ~It covers a¥duF 

\^300,000 references per y ear and- during this time period", 

■groups them into 5 supersections 'composed^ of 80 sections , 

as illustrated in *Figure 4. Each of thfe 80 sections is • 

> ■ • c ' . " 

further subdivided into s.ubsections . . There is 'a total of 

, • ' ■ ■ «• ■ ■ . 

about 700. subjections,, individual records ar:e assigned to • 
categories ^ ^esL fit. Cross -indexirig ternis indicate when , 
other assignments^ were considered accepjtable. C0MPENI)EX ha.s 
f similar structurer' in that each recrod is assigned to cate- 
gories (Card-Alert-Codes). ' However, .the .pbdes are applied 
more in the spirit of controlled indexiTig, and multiple code 
assignments, ^o a given record isr the rule, rather" than the ' 
exception. This is opposed to the fact that a given record 
in CACon is* usually assigned to only one sectipn-and usually " 
has no cross- indexing terms. • ' 

'Each recorid in CACon contains the following fields: COD^N, 
titlfe, indexing <including section ah.d subsection assignment), 
bibliographic reference and author. The plust'ering experimenjts 
used the first three of these fields', iii various combinations. 
The dOMPENDEX records contained the same fields as CACon, and;.^ 
in addition, also coi^tained^ full text abstracts. Clustering 
.experiments "for COMPENDEX used the abstract field. " 
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ABSTRACT SECTIONS 

Biochemistry Sections 

^ 1, Phannacodynamics (CBAC) .... 
2. ^-^Jfonnone PharmacoIoRy (CBAC), 
■ Biochemical Interactions (CB AG) 
4 ToxicoIo:gy (CBAC).. 
^ o-^Agrochemicals (CBAC). . 
''^iT'^enQral Biochemistry. 

T^nzyines . .,. ^ 

>^ Radiation Biochemistry 
9. Biocheiiiical Methods ' . . . . 
1'^ Microbial Biochemistry. . 

U Plant Biochemistry . , 

^2. Xonmammalian Biochemistry, i 

MamniaJian Biochemistry . 
t4. Mammalian Pathological Biochemistry. 

15. Immuiiocheniistry 

16. ^ Fermentations , 

17. ^ Foods : 

18. Animal Xutrition 

19. Fertilizers, Soils, and Plant Xutrition. . ^ 
^0 History, Education, and DociinieiitiiUoli / 



Organic Chemistry Sections/ <^ 
21. General Organic Chemistry 

Physical Organic Chemistry. . . . 

Aliphatic Compounds 

"Alicyclic Compounds . . 
Noncondensed .Aromatic Cotnpounds . . 
Condensed Aromatic Compounds. . 
Heterocyclic Compounds (One Hetero Atom i 
Heterocyclic Compounds (More Than One Hetero 

Atom). ' 

Organ ometallic and Organometalloidal Compounds. 

Terpenoids 

.Alkaloids 

Steroids 

Carbohydrates., , . 

Synthcsisof Amino Acids. Peptides, and Proteins . 



22 

24: 
25. 

28. 

20 
30. 

:n 

32 
33 
34 



^>230<> 
228f;9 
23020 
23179 
23.^79 
23.i.-)4 
24054 • 
24329 
24384 
J4534 
24704 
24084 
25044 
-6404 
2560C 
25869 
2504f) 
2f)00G 
26182 
26327 



25367 
26376 
26605 
26780 
268:38 
270g8 
27034 

27128 
J7300 
27387 
27405 
274J3 
27437 
27456 



ABSTRAa SECTIONS 
^acromoleculor Chemistry Sections (POST) 

35. Synthetic High Polymers i;?7r)13» 

36. Plastics Manufacture and Processing 1376i:i 

37. Plastics Fabrication and Uses 137722 

38. Elastomers, Including Natural Rubber 13778;{ 

39. ^ Textiles ; 137842 

40. Dyes, Fluorescent Whitening Agents, and 

Photosensitizers i:i70l7 

41. Leather and Related Materials 137038 

42. Coatings, Inks, and Related Products 137040 

43. Cellulose, Lignin, Paper, and Other Wood Products IjiTO^rt 

44. Industrial Carbohydrates 138U12 

45. Fats and Waxes. ^ , . . * 138(>l/> 

.46. -.Surface-Active Agents and Detergents 138016 

Applied Chtmittry and Chtmical Englnc«rins Sections 

47. Apparatus and Plant Equipment'. 13802'6 

48. Unit Operations and Processes.", ! 138001 

49. Industrial Inorganic Chemicals ; 138246 

50. Propdlants and Explosives 138347 

51. Petroleum, Petroleum Derivatives, and Related 

Products 138353 

52. Coal and Coal Derivatives 138375 

53. Mineralogical^nd Gcolpgical Chemistl^-. . , i:Wlo 

54. ^ Extractive Metallurgy 1387fiO 

55. Ferrous Metals and Alloys 138851 

58. Monferrous Metals and Alloys 13*K)r)2 

57. Ceramics 1.-10.343 

58. Cement and Concrete Products 1S9424 

59. Air Pollution and Industrial Hygiene 130460 

60. Sewage and Wastes 130514 

6h Water... 139574 

Essential Oils and Cosmetics 130607 

63. Pharmaceuticals 130627 

64. Pharmaceutical Analysis.^. 130663 

PhysiccI and Analytical ChQmlstry Sections 

65. General Physical Chemistry . 130675 

66. Surface Chemistry and Colloids 1.30045 

67. ^'Catalysis and Reactipn Kinetics » 140020 

68. Phase Equilibriums, Chemical Equilibriums, 

, and Solutions.^ I _ , . J40iQ5 _ 

69. Thermodynamics^ Therrfocheraistry, and 'Thermal 

Properties /...' 140288 

70. Crystallization and Crystal Structure. . . ; 140.1.' 1 

71. Electric PhcnoraeUffT 1405;Wi 

72. Magnetic Phfcnomenai • 140878 

73. Spectra by Absorption, Emission, Reflection, or 

Magnetic Resonance^ and OtherOptical Properties 141022 

74. Radiation Chemistry, Photochemistry, and 

Photograj)hid Processes '. . 141444 

75. ; Nuclear Phenomena....! " 14155(> 

76. Nuclear Technology 141I08<> 

77. Electrochemistry 142^89 

78. Inorganic Chemicals and Reactions 14238(5 

79. Inor|;anic Analytical Chemistry 142475 

80. Organic Analytical Chemistry ... 142667 



Figure 4A. CACon Data Base Structure - Section's 
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SubsecUon Arrangement for CA23 - Aliphatic Cocipounds ' 

.0. Review 
!• General 
2'. Hydrocarbons 
3. HaLides. 

^ • ^^^=!^^'r-^ixi€r'o^ . imines , quaternary 

ammonium compounds 

5. Hydroxyl amines, hydrazines, azines, triazines, 
azides, azo and di^zo compounds '» 

6. ' Kitro and Mtroso Compounds 

7. Alcohols and thio alcohols 

8. Alcohol esters with inorganic acids including ' 
cyanates and isocyahates • . • 

9. Ethers and thio ethers 

10. Peroxides, and hydroperoxides 

11. Sulfoxides, sulfones and sulfonium compounds 

12. Sulfenic, sulfinic and sulfonic acids 

13. Selenium and telluritim 

14. Aldehydes and derivatives 

15. Ketones and derivatives 

16. Carbonylic acids and peroxycarbonyli-c acids and 
their sulfur-containing analogs aftd salts 

17. Esters, lactones, anhydrides/, - acyl peroxides,)"*^ 
acyl halides f 

Amides, lactams, amidines. imidic esters, 
(hydr) azides ' , " • 

-Nitriles. isonitriles and acylcyanides 
-20. "Ureas, carbonic acids, guanidines. and sulfur ■ 
containing analogs 



Figure 4B. CACon Data Base Structure - Subsections 
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ORGANilZATlON INDB^ 

> 

Ciyil - Environmental - Geological - Bioengineering 

Planning, design, construction and maintenance^Hixed structures and facilities/ in- 
cluding public works, for commt^^ environmental control, housing, 
industrial actmly^and-transfiorfationr 




Oroup Division 
No. No. 



$ AnnudI 
Subscription 



400— CIVIL ENGINEERING, 
GENERAL 

401 - - Bridges' and Tunnels $65* 

Design, construction, mjinten^nce^and repair of 
jfch, bascule, cable-stayed, cannlever, compos- 
ite, lift, movable, plate gnder. pontoon, suspen- 
sion, swing, trestle, truss and other types of 
* bridges of concrete, masonr/, steel apd niher 
materials tot caiiscLway. highway, military, pe- 
destrian, pipeline; railroad and viaduct appli- 
cations, bridge anchorages, decks, piers, super- 
structures, and supports, construction of pedes- 
trian, railroad, utility, vehicular, water supply 
and other tunnels. 

402 — Buildings and Toj^ers $100 

Deitgn, construction, service equipment, main- 
tenance and repair of apartment, auditorium, 
comftieccial. edutational. exhibition, factory, 
tarm. ga/agc, industrial, laboratory, medical, 
office, public, recreational, religious, residential. 
>l4Gium. stores terminal, theater, warehouse and 
other bciildings; conventional, inflatable, modu- 
lar. muUistory, portable, prefabricated, tempor- 
ary and other tvpes of building construction, 
exposition ^ructuies, masts, monuments, pylons, 
sifos, stacks, to\iers and other special slrjctures. 

403 — ^"Urban and Regional Planning 
ahd Development $65 

Design and development of urban areas and 
* regwins including cities, suburbs and towns; ^ 
lan^ use planning; municipal engineering and 
pubiic works including provision o» facilities 
and structures for education, government, health, 
hot'^ng, recreation, shopping, and urban trans- 
port including internal transport facilities, urban 
rehabilitation and renewal 

404 — Civil Defense and Military . 

Engineering $65 

Cmhan protective works and shelters, military 
bases, buildings, construction, equipment and 
Riatcrici, military research on ballistics, missiles 
and other ordnance- 'military science, missile 
^ifes and s>^i ems; naval bur'dings and structures 

405 — Construction Equipment and 

Methods; Surveying $100 

Design and manufacture of blasting equipment, 
caissons, cofferdams, concrete mixers, construc- 
tion vehicles, cranes, derricks, dredges, earth- 
rroving equipment, 'hoisting equipment, piles 
and Rile drivers, pneunatic tools, power shovels 
and other equipment items, construction opera- 
tions such as dredging, erection, excavation, 
grading, grouting, masonry, prefabricated con- 
struction, riveting, rock drilling, and shaft sink- 
ing technfques of concrete, steel, and timber 
construction. tcch.iK|Mcs of surveying and ma'p- 
ping.. including pivotogra'mmelric methods 

406 — highway Engineering $65 

Mtghways roads and streets engineering includ* 
ing culverts^ drain^e, embankments, inter- 
changes intersections, lighting, markings, me- 
dian dividers ^nd guard rails, overpasses and 
underpasses, railroad* crossing;, road stabiliza- 



Croup Division 
No. No. 



$ Annual 
Subscription 



non and structural design, roadside improve- 
ment, route planning and siting, toll roads and 
related structures; maintenance of highways and * 
ether routes, 

407 — Maritime and Port Structures; , 
Rivers and Other Watervyays $6S 

Design, construction, equipment, maintenance 
and repair of breakwaters, docks, groins, fetties, 
marine terminals, piers, pontoons, quay walls, 
revetments, seawalls, shore and harbor protec- 
tion and coastal engineering structures generally, 
harbor and port facilities, lake, river and other 
waterway improvement and regulation by means 
of dredging, navigation canals, channels, gates 
and locks; sedimentation and silt control, and 
bank stabilization. 

408 — Structural Design %100. 

Design, construction and testing of arches, 
beams, columns, cylinders, disks, domes, framed 
structures, girders, plates, sheet materials, shells, 
spheres, struts, trusses and other structural mem- 
bers, sections and shapes; structural stress 
analysis, photoclasticity and other methods of 
stress determination in structural design, wind 
stresses. 



410 — JCONSTRUCTION 
MATERIALS 

411 — Bituminous Materials $65 

Manufacture, testing and use of asphalt, pitch, 
tar anu derivative byproducts for applications 
^ such as coatings, flooring, pavements, roads 
and streets, »oofing, sealants and waterproofing. 

412 — Concrete $100 

Admixtures, aggregart^s, cement, crushed stone 
gravel, lime, mcrtar, ready mix, reinforcing 
materials, sand and combinations thereof to form 
concrete products, lightweight concrete, rein- 
forced structures and surfaces irltluding blocks, 
precast and prestressed units and other structural 
forms. 

413 — Insulating Materials $100 

Asbestos, cork, fiber and fiberbdard, foam ma- 
terials, glass, magnesia, mica^ mineral wool, 
plaster and plasterboard, plastics, rubber, ver- 
miculite^ wax and other insulating materials as 
used for acoustical, electrical, flame, moisture, 
radiation, reflective, sound, thermal, and vibra- 
tion insulation. ^ 



Croup 
No. 

420- 



Division 
No^ 



$ Annua! 
Subscription 



414 — Masonn^ Materials 



$65 



Baolt. brick, clay, glass, granite, limestone, 
marble, sandstone, .slate, terra cotta, tilc and 
other structural ceramic and stohe materials for 
buildings, engineering works, arid structures, 
mortars;* 

415 — Metals, Plastics, Wood and 
Other Structural Materials $65 

Aluminum, copper, iron, magnesicm, plastics, 
slecl„ wood, and other structural materials to 
form clad, composite, honeycomb, laminated, 
reinforced or sandwich materials for building 
and structural use. 



MATERIALS PROPERTIES 
AND TESTING 

421 — Strength of Materials; 

.Mechanical Properties $100 

Elasticity, plasticity, rheqiogy, stress-strain rela- 
tions and associated phenomena and properties 
such as abrasion resistance, crack formation, 
creep, deformation, ductility, failure, fatigue, 
fracture, hardness, malleability, radiation dam- 
age, strain hardening, strength, surface rough- 
ness, wear, yield strength and other mechanical 
properties, testing of metals in bulk form or 
as crystals, films, foils, sheets, whiskers, wire 
and powder meU\ products; testing of nonmetal- 
' licj in bulk or divided form or as combinatinns 
of materials such as composite, honeycomb, 
laminated, reinforced and sandwich materials. 

422 — Strength of Materials; Test ^. 
Equipment and Methods $100* 

Apparatus such as hydraulic impact (e.jj. Charpy, _ 
Izod), inden'tation (e.g. Srinell, Rockwell, Vick 
ersj, screw-gcar ofld universal machines, aim 
instruments such as exicnsometers, strain gages 
and other crevices; bending, compression, creep, 
iatigue, hardness, high and low pressure and 
temperature,, impact, shear,' tension, and torsion 
test methods, nondestructive teCf^iques such as . 
brittle coating, liquid penetrant, magnetic par- 
ticle, radiographic, ultrasonic. X-ray and similar 
means for detection of defects and flaws; special 
techniques for accelerated lestirig. 

423 — Miscellaneous Properties and^ 

Tests of Materials $100 

Other physical and general properties of mate- 
rials as determined by miscellaneous test equip- ' 
ment inc'uding chemical, electrical, envjron- 
meiital, nuclear, optical, physical and thermal 
apparatus and instrumentatipn. 



430 — TRANSPORTATION 

431 — Air Transportation $65 

Air cargo, freight, mail and passenger services, 
civil and milifary; aircraft maintenance and 
repair facilities and'' methods; airlines, reserva- 
tion systems, routes, scheduling, airports, build- 
ings, hangars and terminals, ground facilities, 
markings, runways, air safety, air traffic control, 
navigation aids. 

432 — Highway Transportation $65 

Commercial, freight, passenger, public service 
and other forms of motor transportation employ- 
ing automobiles, buses, taxis, trailers, and trucks 
and including operation of fleets, lines, routes 
and terminals; filling stations, garages, repair 
shops and vehicle maintenance and repair, 
^ highway safety, traffic control, signals and 
surveys. 

433 — Railroad Transportation $65 

Freight and passenger rati se^^ices and industrial 
railroads Including use of rail-highway containers 
and trailers, and operation of lines, reservation 
systems, routes^ switchyards and tCimmals; re* 
pair shops and maintenance and rcpitr of rolling 
stock; safety, signal systems and traffic control. 
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434 — Waterway Transportation $6%^ 

Cargo shiprrsnl jnd passenger f^.oportation on 
coastal, inland, transoceanic or other routes; 
cargo transfer and terminal operations; marine ^ 
safety and navigational ' aids' including beacons, 
buoys, * lighthouses, lightships, operation of 
barges, containership<, ferries, freighters, me"' 
chant ships, passenger vessels, tankers, tugi and 
other craft. 



440— WATER AND WATFP.WORKS 
ENGINEERING 

441 — Dams and Reservoirs; Hydro 

Development $65 

Design, construction and repai- or arch, buttress, 
earth, embankmeni, gravity, movaDle, and rock 
till dams, multipurpose and special purpose 
reservoirs, hydraulic structures associated vvlth 
dams^ and hydrO'power develop.':;ent such as 
channel and chutes, conduits, draft tubes, fjsh- 
* ways, flumes, forebays, penstocks, river basin 
development, siphon^) sluice gates, spillways, 
^ stilling basins, surge thanks, ancf weirs. 

442 — Flood Control; Land 

Reclamation . $65 

V 

Drainage, runoff and subsurface water quantity 
control; flood routing, flood ccniiol measures 
and structures such as dikes, drainage basins, 
ievees, river embankment work^ 2nd storage 
systems, flood forecasting, m9^sure<, structures 
and v^orks for irrigation and reclamation of land 



443 — Meteorology 



Aerology, agronomy, atmosphere, climatology, 
cloud lormatam and seeding, rce, ram, snow, 
and stoyn phenomena, weather modification, 
winds, weather forecas^ng and m'^^jurement by 
ancmometric, barometric, hygrometric, pressure, 
temperature and other instrumentation including 
use ot meteorological balloons, radiosondes, ram 
and snow gages, utelhtes and telemetry systems 

444 — Water Resources $65 

Suriace and underground water occurrence, re- 
sources apd supolie^ including aquifers, artesian 
water, groundwater, springs, water bearing for- 
maiionf zhd strata, waterfalls, watersheds, water 
wells, and hydrogeology, water conservation, 
water law, water prospecting, w-Icr yield im* 
provc^cnt, regional water resources, hydro* 
log(C-al cycle generally induding evaporation, 
precipitation and transpiration of moisture and 
. its influence on atmospheric water va{^or, soil 
mois'ure, surface water and water table, regional 
hydrology 



450 — POLLUTION, SANITARY 
ENGINEERING, WASTES 

451— Air Pollution $100 

Engineering and economic aspects of air pbllu- 
tion control; abatement and control of gaseous 
and particulate polliitai^ts such as dust, engine 
exhausts, flue gases, fly ash, fumes, odors, smoke 
and soot; methods and equipment usH for air 
and dust analysis, density measurement and 
sampling; dust collectors, filters, precipitators 
and recovery" systems; dust haiards and pro* 
tecllve devices. 

452 — Sewage and Industrial 

Wastes Treatment ' $100 

Environmental sanitation prairtices, particularly 
the disposal, removal and treatment of agricul* 
tural, community and industrial sewage, design 
and development of incinerators for conversion 
and disposal of soiid wastes, recovery .of thermal 
energy^ recycling and production of' useful by- 
products; design, construction, operation, main; 
tenance and repair of sewage treatmon: plants 
including equipment such as filters, pumping 
plants, pump^ and tanks; sewers and street 
sanitation. 

453 — Water Pollution $65- 

Abatement and control of biological, chemical, 
physical, and thermal pollution of shores, 
streams and' waters generally by industrial pro- 
, cess effluents, mine drainage, qatural ct^trophi- 
cation, oil spills, radioactive materials, refuse, 
salt water. intrusion, sewage, wastes and other 
pollutants. 



$100 460 — BIOENGINEERING 



461 — Biotechnology 



$100 



Engineering aspects of human factor »^uire- 
ments in the design, development and operation 
of man*machine systems; biomechanics, bio- 
medical measurements, biometrics, ^bionics, 
cybernetics, ergonomics, and |ifc-suppon sy'iems 
generally. 

462 — Medical Engineering and 

Equipment $100 

Devices and instruments for nr\edicaf practice 
and research including equipment for specialties 
such as anesthesiology, cardiology, encephalog- 
raphy, fluoroscopy, instrument patient monitor* 
ing, radiology, and surgery; -design and manu- 
facture of hospital equipment and facilities; 
design, manufacture and materials for use In 
medical supplies such as artificial organs, car* 
diac pacemakers and valves, dental' materials, 
eyeglasses, hearing aids, prosthetic devices, 
\spirators and therapeutic aids 



ods, dnllmg arid sampling, exploration, labota» 
'tories, ocean floor mining ^nd research, under- 
water ^ life>support systems and spcialUed 
equipment; use of diving and salvaging appar* 
atys, ^submersibles and undfersea vehicles &nd 
systems'^ / 



48Q,— ENGINEERING GEOLOGY/ 

481 — Geology and Geophysi'cs/$100 

Engineering aspects of earth sciences»'^lnciudlng 
economic geology', geological dating, geomor*' 
phology, physical geology, regional g(jo!ogy, 
sedimentology, stratigraphy, structural geology 
and tectonics; factors affccting*constructibri and 
- ■ location of engineering works due to ge9iogtcal 
conditions, geochemistry, geothermal phenomi^ 
ena, and terrestriaf electricity, magnettsm and 
physics including properties of lonospqere and 
upper atmosphere generally of gcf^physical 
interest. I 

482 — ' Mineralogy and Petrologf $100 

Chemical and physical properties, classification, -^ 
composition, crystallography, formation, nature, 
occurrence, origin and use of minerals/ occurring 
naturally including precious and semi-precious 
gems, rocks 3nd^,stones, lithology, petrography 
and petrology generally; regional miperalogy. 

483 — Soil Mechanics and ' 

Foundations ^ $100 

Design and construction of foundatiiins and soil 
structures related to engineering^ w|l}rks suclr as 
buildinc^^am sites, earthwork^ embankments, 
and ear^^otaining structures; investigations and 
soil surveys by means of boreholes, sampling 
and othsr techniques^ properties oj clay, gravel;, 
muskeg, permafrost, sand and silt; grouting, 
soil compaction, consolidation ana stabilization, 
testing and evaluation of such rWcchanical and 
physical properties as bearing Capacity, perme* 
ability, strength, and trafficabifity. 

i 

41B4 — Seismology ; $65 

Analysis, recording and study jof earthquakes, 
microseisms and other seismicj action due to 
earth disturbances and volcan«j eruptions, de- 
sign of earthquake resistant Structures; land* 
slides, isunamis and other secondary effects of 
earthqual^cs, seismic stations, seismographs and 
seismometry. | % 



1» 



445 — Water Treatment/ Genera! 

and Industrial $65 

Improvement of water quar:^ for general, pota- 
ble or process use; methods and equipment 
designed for aeration, chlormation, coagulation, 
demincralization, filtration, flocculatlon, fluori- 
nation, sedimentation, softening and other treat- 
ment techniques, water Analysis, bacteriology, 
and chemistry; saline water conversion 



446 — Waterworks 



$65 



Design, construction, equipment, operation, 
maintenance and repair of water supply systems 
including aqueducts, distribution lines, mains 
and water pipelines generally, municipal water 
supply and regional waterworks; pumping plants 
and stations; water tanks, lower» and related 
hydraulic structures; water utility management. 
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470 — OCEAN AND UNDERWATER 
TECHNOLOGY 



471 — Marine Science and 
Oceanography 



$100 



Chemical and physical properties qf seawater, 
currents, tee formation, tides, waves and weather 
effects, and engineering implications; island 
formation and erosion; ocean bathymetry and 
hydrography; sea as source of chemicals and 
minerals; sea as source of food, induding fish- 
eries; equiprnerit and research. 
>• 

472 — Ocean Engineering $65 

Submarine geology and geophysics: undersea 
region a< environment, habitat and sea bed re* 
source; under^cji chambers, construction meth' 
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CLUSTERINp ALGORITHMS 



The mathematical steps r;equired to construct clusters 
are simple. \ One way to do it is to define the distance between 
all pairs of^ records by the equation: . ' ' ^ 



r 



'\^ere^^ D(Ri,Rj) = Distance between records i 'arid j 

\ N(RinRj) = The number of teirms in common 

\ between records i and i . 

•.\N(RiuRjO = The number of terms in either 



•1 or J 

\- 
V*. 



This distance is known as the Tanimoto or Jaccard distance. 

qi early," this equation satisfies' the . intuitive notion of distance. 
If records i and j have all their term's in common, the distance ' 
between them is/ zero. \ If: records i and j have no terms in common, 
the distance between them is the maxibum, 1. Thus the distance 
between records is just\ a measure of the term .overlap between them 

One possible procedure for using the distance measure to 
partition the retrieval §et is to find the distance.? between 
all pairs of recprds, 'and^ then to join, into clusters those 
records- that are separate^ by the smallest distances . That is, : 
join the closest pair, the^ the next closest pair, etc„ until., 
only a manageable number ' of groups , about 20, remain/ Jlaliy 
'variations on this theme, haWe been .tried by various research 



groups . ^ ^ 



All experiments in fchi^; study were performed using a 
variation- of this procedure 'palled the Lance and Williams "Group 
Average'' algorithm..^ 3 . 2 xhis selection" was based.on sevferal 
factors.' First, since the clustering was only t6 be-' applied to 
small files, algorithms that depend on N^^ instead of the less 
expensive In in their space and time Requirements "could be 
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afforded. Second, the Lance and Williams algorithm can readily 

• b*e 'modified to accept distance thresholds, statistical , term 
weighting and multi-stage processing. ^ Following Van Rijsbergen^^ 
it: has been found that most measures yield nearly equivalent 

'results since they use the same information. The steps -to the 
algorithm are: . ^ ' 

1. Calculate tfhe distances between each pair of records. 

2. Select the two closes't entities (either single, records • ' 
or clusters) and merge them to form a new cluster. 

3. Calculate the distance from' the new cluster to each 
remaining entity. 

4. If m6re than one entity is left, 'g.o back to 2. 

' ■ •. « 

The calculation i'vf Step 3 is as follows: 

If record i and record j iiave b.een merged, to form entity x, 
and the distance between record i and record -j is denoted 
D(Ri,Rj), then for all entities a; ^ ' 

\' Or 

, D(q,x>H= N(RiIlD':Ri.q) ^ N(R1) .Dgil^g). 

N(Ri) + K(Rj) ^ . . * 

• where N(Ri) is the number of records in entity. Ri, which is 
one: ■ Similariy, N(Rj) = L. , and N(x) = N(Ri) + N(Rj) = 2. 

I , 

This is, tna.i, an agglomeratiye method. The clusters grow by . 
fusion until the entire corpus forms one cluster. The corpus 
can the be dividfed into ""0- clusters/' by taking all the clusters- 
farther apkrt than p.. .. The distance betweeh any two records can 
be defined as the distance at which those two records are first . . 
joined in one cluster. ' . 

The result of this sort of clustejingjis generally represented 
'by a tree structure, called a' (5endrogram, in which each record is 
represented Sy a- leaf. Nodes, if/ the dendrogram, representing 
joined rei,ords, are formed a4: characteristic distances. The 
distance between two records is the distance at which they are 
first joined (See Figure 6). 
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R"i . Ri ^ Ra Rit • Rs 



Figure 6. Prototype Dendrogram 

In most dendrograms nodes will occur at several different 
distances between 1 and 0. 



Lacking a plottitig device, computer generated dendrogram 
representations had to be reformatted somewhat to be suitable 
for display (See Figure 7) 

Though the previous discussion has been concerned with 
the clustering of records; it is often useful to cluster 
terms, thus building groups., of "synonyms". This can be done 
using exactly the same algorithm as before. Just as a biblio- 
graphic record can be treated as a list ,of terras 'tg be clustered, 
the inverted file of postings that is associated with' a single 
term can be treated as a list to be' clustered. The equivalence 
of those 'procedures is indicated graphically in Figure' 8; * 
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CLUSTER DISTANCE 



RECORD 
NUMBERS 
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Note:' for example, that records 13 and 39 are joined at a distance 
^ of .25. Similarly, records 3- and- S.are joined at a distance of .33, and 
so have less term overlap than do records 13 and 39. . 

Figure '7. Sample Denaogram 
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RECORD 1 

Ter^i 1 

.Term 7 ^ 

Term 8 

Term 26 

Term 147. 




RECORD 2 

Term 6" ' 
Term ,8 
Term 26^ 
Term 35. 
Term -104 



a. Record Clustering 



/ 



TERM 1 

Post 8 
Post' 17 
Post 108 
Post lie 



TERM 2 




b. ■ Term Clustering 



'The same algorithm Lhat clusters records over their terms (a) 

■j ' • 
can cluster, terms over their postings (b) (Links shown) . 



Figure 8. Relation Between Record and Term Clustering 
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MEASUREMENT OF CLUSTERING gARAMETERS 



^ Three key parameters .characterize' the usefulness of a ^ ^ 
cluster ruIJ^ * ' . ' • - 

. 1, "the traction t)f the file that is allocated to groups^ 
/ '.'(coverage)., , ' . 

2. ' The average siz^ of the groups formed (agglomexBtion) , 

and • . \.' ' ' ' ^ - , , \ , 

3. *The fraction .of the file that is allocated correctly. 
\ (accuracy). 

These patameters are evaluated according" to the following rules. • 

Coverage: Any recx)rd " is counted as clustered at a distance 
, ' , D if it participates in aplea'st one join with 

another record at. any distance less than or equal' - 
^ . to; D. ■ . ■ ■ / . ' 

Agglomeration: Agglomeration (N^) is measured as the average 

size of the clysters that are formed at a -Si^tance 
• D. It' is calculated as the niimber of records 

clustered (N^,) divided by the number of clusters '(N) 



N 



- Accur<acy: If records -of two kinds (A and B) ara clustered 
(at a distance" P) , a cluster is counted as being 
* of the A type- if the majority of records, in the 
cluster are A- type, ^and as B if they are o^ B type. 
The A records' in an A cluster are counted as 
, • correct, and the B records in a B clus^r ^re counted 
as correct. Conve'rsfely, A record? ir^ a B cluster,'- 
or B records in an A" cluster,. are counted, as, . , 
incorrect assignments. If there arfe an equal ntimber 
of A and B records in a cluster, then half of the 
'. total are counted as correct. 
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3. * EXPERIMENTS ^ ' ' 

, In- order t<5 te^ the feasibility of. the sequential 
processing hypothesis, 4 experiment were conducted.. Bach 
experiment was designed 'to answer a specific question abo.ut; 
the limitations' of statistical string .processing. 



\* < 
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EXPERIMENT 1 , , ' 



The .question^ addressed by the first e^jcperiiqent is, "Can 
direct vocabulary feedback to a sea'tcher adt as* a. useful 
summary level device?" Th$it is-, in seeking a. mechanism- to 
characterize a retrieved set, it is natural- to consider a . 
-sorted Lis t^ of the terms .present in the records. Current 
on-line systems provide some vocabulary support, such'as ' 
listing terms present^in' the fiata base that are alphabetical-' 
ly close j:o .a given term 'or delated to a given term by ' • 
subject content (broader temiy narrower terjn, synonym, etc.) 
However, the information given by this vocabulary support 
capability applies to an entjLre data base rather, than to a - 
retri ved set. ^That is, one can 'readily obtain *a sorted list 
of the terms present* in 'the -whole data base,^t)ut not the terms' 
in a retri^eved set. . ^ - ^ 

Since the searcher evaluates records by looking f^r oc- 
_curreiices of terms, it seems natural to -have the computer 
simplify the task by presenting to the. user sorted ;is't of 
the terms present, in' tHe initjLal retrieved set. Vn this 
experiment , it was found that the number of terms on which'^the 
relevancy decisions are based is usually jus^ a few ^percent 
of those tenns present (though the set o'f crucial terms may be 
different for different users even if they aire concerned with 
the same initial retrieved S-et^ . Thus, i>: is appealing to 
consider how the terms might be sorted "for "feedback". 'Some ' • 
sorting is necessary, as even for a mere 100 records there 
are about 1,^000 unique terms in the .title and .index i^ield for ' " 
CACon - too many for the' user to benefit frbnj having tbrysc'an- 
all of them rather than the entire retcords.- It was cotijectured 
chat simple frequency criteria might be sufficient to identify 
the key- terms. Tor this experiment, typical retrieved sets,' 
containing relevant and non-relevant records • (about 50 of each 
type) WQre characterized by the tetms that they contained. The 
crucial terms) on which the relevancy decisions were based, were 
identified. It was .found that they could not be identif^.ed by^^ 
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simple statistical criteria • Often, 'low frequency terms were 
crucial when tji^y indicated specific concepts that were not 
relevant. However, in 'other cases, High frequency terms were 
necessary/.^ The inability of gross frequency data to sele'ct 
terms 'appropriate for searcher feedback led to the postponement 
of consideration of vocabulary feedback until after vocabulary 
mapping experiments had been/ completed (Experiment 4)- T^e 
vocabulary -mdpping involved semantic input and promised to in- 
crease "the efficiency of retrieval abov^ 'the level of purely 
frequency-based criteria. The possibilities of vocabulary 
feedback based -on this semantic input ins^tead of gross frequency 
data axe discussed' further in Section 3. ' 
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^ gXPERIMElx'T 2 

* I • * 'ft 

The following questions were addressed ^by the secorid 
experiment. Can clustering resolve record classes with 
.substantial vocabulary overlap such as will occur as the 
result of a Boolean retrieval? How does the resolution 
depend on the mathematical details of the clustering pro- 
qedur^ What are the relative contributions of the various 
record fields (title, index, abstract^, CODEN) to re^^ution? 
That is, clustering can be expected to easily Resolve re- 

' cords from disparate disciplines into separate groups, in 
cases whete overlap bet\>,een the two disciplines is-^mall,. 
such as high .temperature physics and botany. It is less 
clear that clustering can successfully resolve records 

' from disciplines with much vocabulary overlap. (See Figure 
9) . 

The design of Experiment 2 is indicated in Figure 10. 
Fifty records were taken from each of two sections of CACon 
or COMPENDEX and were put into one file of 100 records. 
CA.Con has a subject organization, so that all the records- - 
contained in a given section pertain to a given subject, such 
as- "Hormone Pharmacology" or "Mammalian Biochemistry". Card- 
A'lert-Codes play a similar role- in COMPENDEX. ^^rhen the file 
with 100 records is clustered, ideally it would divide into 
twQ^lusters, ^ach^j^t^afnitTg 50 records 'from one section. 

'Soffie'typrcal results are shovm in Figures 11 and 12. 
When the two sections used are disparate in subject '^rea, such 
as -the sections on "General Biochemistry" s^d on "Terpenes", 
the separation achieved closely approximates* the ideal when 
the title and index fields are included. 

When the two sections selectedj:have;,greater vocabulary 
overlap, such as the sections "Terpenes" and "Carbohydrates", 
the separation is much less successful. \ A number of generaliz- 
ations can be dra^ from the data. In an effort to measure the 
effect of the mathematical details on the separation, several 
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different clustering procedures were tried. In general, it was 
found thd't the piroblem lies mostly in the s.tructure of language 
> not in th| mathematics of classification. Th^t is, the 
experiments suggest agreement with Van Rijsbergen^ s , that most 
measures yield similar results because, ultimately, they are 
based on the same infcJinnation. . Also, it seems that further 
improvement requires additional preprocessing, such as genera- 
tion o£ a degree of semantic structure for the vocabulary. 
Clustering without any additional vocabulary preprocessing will 
be called simple clustering. \ 




A B c 



Figure 9. Effect of Term Overlap on the Resolution of Record 
Groups 

To be useful as a second step retrieval device, clustering 

must function well in Case C. 
» 

Clearly, most algorithms can separate grbups such as A 
wherein term overlap is negligible. Seiparation is mor^ diffj.- 
cult for B, wherein term overlap is slight but non-negligible. 
Separation for Case C is required if clustering is to be suc- 
cessful as a second step search mechanism, for the* set selected 
by the first search step x^ill have much overlap, as all members 
were selected by a search strategy. 

The results of Experiment 2 suggest that records from 




different supersections of CACon areUike case A and are easily 
separated* Most records from tWQ different sections are like 
case B and are separated with acceptible efficiency. ^However, 
records from related., sections are like case C and arel'not 
separated acceptably by* simple clustering. Since. case C cor- 
responds to the kind ,of .overlap -foimd for record sets retrieved 
by a Boolean search, it seems unlikely that simple clustering 
can partition retrieved search sets into relevant and non- 
relevant clusters with acceptible efficiency. 

The surprising result that the inclusion of the abstracts 
field made only a small contribution to record resolution by 
simple clustering is related to the effect of high frequency 
terms on the pattern, and is discussed in Section 4. 
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Experiment 2 



50 Records 


from 


Section A 






50 Records 


fr^rn^ 


Section B 






100 tRecord'W 
Combined Fi 

L 



/ 



Records 
from 

Seption A 



Records 
from 

Section B 



Results: ^ 



Effect of variation in cluster algorithm 



I Using only non-singular terms" improves 
cluster separation 

i Details of the distance measure seem to 
have only a small effect on the partition 



« ^^Rffect of different data fields on 



the partition 



I 

i 
B 
I 



CODEN field is useful | 

Index field is the best ' 

Title fields is second best 1 . 

Abstract field makes only a smkll contr,ibUtion 



Effect of section choice on accuracy of partition 

■ Records from sections characterized by 
very different vocabularies are^. easily 
distinguished \ 

■ Records from sections^Kafac'teri^sed by 
similar vocabularies are not easily 
distinguished 



Figure 10. Design and Cone ?i.ust ions for Experiment 2 
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Typical Results: CACon rSections on General Biochemistry 
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^Results for CACon Sections on "Terpenes and "Carbohydrates" 
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Figure 11. Typical Results for Experiment 2 
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Figure 12. Experiment 2 - General Summary of Data 
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EXPERIMENT 3 . • • - - 

The -"question addressed by the third experiment is; "Can 
simple clustering separate the user-judged relevant records 
from the non-relevant ones?" The experimental procedure is 
« illustrate^! in Figure 13- Searches performed by IITRI's' 
Computer Search Center and evaluated by users in the normal - 
course of center operations were used as the basis of* the 
test. For each of the experimental tests, fifty relevant and 
fifty non-relevant records, for one tiser, were put together in 
one file of 100 records. Then, the file was clustered. Again, 
as ixi Experiment 2, the ideal condition would be to have -two 
clusters formed, one with 50 relevant records and the other 
with 50 non-relevant records. Results indicate that although 
the separation produced by simple clustering is not good 
enough for it to serve as a reliable high-precision second 
step mechanism, it does approach an acceptible level in many 
runs. Hence, motivation was high to explore the structure of^ 
vocabulary and its implications in. the fourth experiment, in 
the hope that the addition of some semantic information would 
increase the second step efficiency to the point that it would 
be immediately practical. ' 
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Clustering assigranents are made with good 
accuracy it small cluster distance^ hut 
not 'at large ^cgjies . ' , * 

The fraction of the file that is ^clustered 

is sufficient oaly at large cluster .distances. 

The average cluster size is acceptible only 
at very large clu'ster distances. 

Simple clustering. is not practical as a second- 
step mechanism for any file configuration tried, 
although results approach practical levels for 
many individual runs . ' ^ ' 

'S^rther progreesNWould be greatly aided by 
incorporation of ^a. degree of semantic informa- 
tion in^the clustering process. 
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Figure 13., Experimental Design for Experiment 
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Ss Three key parameters ' specify the usefulness of a cluster 
run, namely coverage,, accuracy and agglomeration. If coverage 
is low, part of the file is not included in the pattern; if 
accuracy is low, the pattern is worthless; if agglomeration is 
low, the number of decpLsions that the user saves is low. That 
is, if there are records per cluster, and ©nXy one of them 
need be evaluated to evaluate all by implication, then (N -1) 
decisions are' 'saved per cluster. If a file of Np records is 
divided into^ group of size N^, then there are Np/N,^ groups 
and the total nijmber of decisions- is reduced from to N^/N . 
Unless is large, .the savings is small. Figures 14, 15 arid 
16 show the summary of these three parameters obtained, as a 
function of cluster distance for 50 Aiser evaluated retrievals 
(each containing' 50 relevant plus 50 nonrrelevant records) 
clustered under the protocol of Experiment Each data point 
Represents the average value of a parameter for the 50 runs, 
and each vertical bar delimits the one standard deviation . 
interval from the average at. that point. According to Figure 
14, ^nly at distances grea er than 0.95^^315^^*^-01 overlapping 
term among two records with 10 terms each) is isubstantially 
all of the -file clustered.- About 80% of the file is clustered 
\t a distance of 0.8. 

According to Figure 15, the number ot records clustered 
correctly is approximately equal to the number clustered at 
small d^istances, but it falls off at high distances. At a 
distance of about ;95, only about 70% of the records are 
clustered correctly. According Figure 16, the agglomeration 
does not becoipe^ appreciable until cluster distances are . 
greater than about 0.9. In summary, simple clustering can \ 
separate relevant records from rion--relevant ones with' suf- 
ficient accuracy 6nly at very small distanc9s, whereas agglom- 
eration and coverage are sufficient only at latge distances. 
To improve upon this situation it was decided, upon surveying 
individual runs for the r.easons of ^clustering failure, that a 
mechanism was needed to allow the relation of nojn-identical 
strings on the basis of their semantic relationships. To that 
end, the vocabulary mapping experiments were initiated. 
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EXPERIMENT 4 ^7 /VOCABtJLARY MAP3!Ng ^ f 

Even before the first 3 *experimeixtjs were done, it was ' 
recognized that there is one major reason why simple clas- 
JterJLng x/ourd not be expected to^work well enough to correct- 
ly classify a colleetion- of records with.^signi^icant 
*vocabulary^ overlap. Any callection of records can be 
classified (ordered or partitioned) in many different / 
intellectual ways. Simple clustering, as described, earlier, 
is merely on'e arbitrary way of classification. As such, it 
is not clear that it should be expected to separate the 
relevant records from the non-relevant or\es or to separate 
^ records into groups that 'are?. meaningful t;o 'a given user ' 
because what is^relevant depends on the- intellectual clas- 
sification priTiciples of the tiser. For example, suppose*- 
that the user entered 'a Boolean search on the" subject of * ^ >' 
plants and air pollution. The resulting retrieval could be 
intellectually categorized according* to the species of plants 
involved, putting, for instance, hardwood tree*^ in one group, 
softwood ^rees in another, shrubs in "another, etc. Alter- 
natively; the records could "be intellectually categorized 
according to the chemical air pollutants involved, SO2 in 
one group ^ NO2 in another, ozone in another, etc.. Similarly, 
the intellectual categorization could be based on w^a'ther 
conditions, .geography, economic, impact country of origin, 
etc. Thus, since the computer at present does not have the 
definitions of the termg, the problem of constructing us^- , 
meaningful partitions has two- levels. First, the system has 
to have a way of homing in on the intellectual principle of 
classification (i.'e. in the sense that dategorizirig the 
example retrieval on the names of ^ the plants involved is an 
intellectual principle of classification). Second, a way 
has to be found to direct the. classification, mechanism 
(clustering) to. use the classifying principles .specified 
by the user. • • , 

The solution of these problems requires that -the system 
has additional 'semantic information available. That is, while 
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the full dictionary- type' definition of each string may not be 
required for processing at this .stage, there must be at least 
ciaw^^h information to distinguish the teiins among the vatrious 
'common- inteljfectual organizing principles /to which they may 
apply. To this end, it has been found desirable to map each * 
term^into a conceptual category. Thus for instance, suppose 
^cbtk were mapped into the' categor^y "plant", NO2 and SO2 were 
mapped into the category/'air pollutant", etc. Then the 
selection of an intellectual principle of classification 
would correspond merely to, the selection of a term category. 
That is, if the 'terms that denote the names of plants were 
labeled as belonging to the class of plant names, they Would 
be singled out^ by *the- computer as the string symbols or: ; 
which to base a record classification even though the com- 
puter could not distinguish amolrg those n&mes any secondai^y 
characteristics (i.e., "tomato" is defined onljr^s a member 
or the class of plant names). So the keV is that to clas- 
the records according to the principle "plant names"; 
one should cluster on only the subset of all the. terms 
present that, pertain to plants. More generally, to classify 
records according to an intellectual principle, cluster' on 
only the terms that are> members of the term class ^ that ] 
corres^ponds to that principle. Since the terms so choseh 
are only a small subset of all thpse that are present* in a 
record, IITRI has named this process Subs'et Based Clustering , 
or SBC. . • — ' ' 

A, secondary advantage o£ constructing term -classes is 
that it of f ers ^e possibili^ of overcoming some of the 
Um.itation of the binary value of string match. For example, 
the' term "dog" and the term "greyhound"' are not -identical 
character strings, and so they do not match. Similarly, the 
terms "bean" and "dog" do not match. Yet, clearly 'Vdog" is 
much more similar to "greyhound" than it is to "bean". One 
way to enable the system to coupute on the basis ofdey -ees ' 
•Of -similarity is to record the terta association probabilities 
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for a body of ^zex-fe-f-ijarfti to make the assmnption tha^t terms that 
tend to occur together, are semantically rel^ated. This technique 
has been used to great advantage by Salton^^^. Unfortunately, 
it is^ expensive to compute, store and access term corrtic^cion 

'coefficients for large data basest This project has^ attempted 
a different approach -based on the definition of intellectual 

.word classes. » » 

One might argue that terms ere defined, by the context in 
which they occur. That is, medical terms .occur ^in , medical 
records, engineering terms in engineering records, etc. Using 
th^Ts idea, one might represent each term by the List of records 
in which it occurs. An initial attempt to overcome the limir 
tation of binary match (matching is either identical 6r zero 
(1 or P))was based on this concept. The idea was to take the 
small record set that^would result from a Boolean .search, and 
to cluster the terms over the records, in effect defining a- 
similarity between terms based on their co-occurrence, within.^^ 
the records of the small Boolean search. Then, the term 
similarities would be used to cluster the records (sequence 
J>hown in Figure' 17) . A typical term map resulting from such ' 
a sequence of operations is,sno\<m in Figure 18. This sequence 
of- operatioBs is appealing because it is inexpensive and selt- 
contained. The clustering of terms i^nvolves only the small' 
set and requires no dictionary loop-ups. Unfortunately, it, 
was found that this, processing sequence laakes only 'a marginal 
improvement to the resolution of record clusters. The essential 
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Figure 18. Typical Term Map Derived bv Procedure of Figure 1/ , 



problem is that defining similavities between terms is essen- 
tially a global property, and it is unrealis.tic to hope that 
the strings can be classified merely on the basis of their 
associations in a small record set. That is, similarities 
are not well enough defined using this method for small ' 
record sets, and for large record set? the process is expensive. 

The process of context definition seemed to be sound, so 
.additional effort was made to apply it on a global scale (i.e. 
to a whole data base). The conceptual -organization of^ke 
CACon data base into supersections , sections and subsections 
(see Section 3) suggested that terms could be characterized 
by thei-.- occurrence in this hierarchy. That is, 'reco'rds 
are filed by CAS indexers .within the CACon section structure 
according to their, intellectual content. Because the 
intellectual content is represented by terms, th?y are im- 
plicitly filing the terms according to their intellectual 
relationships. Accordingly, it should be possible to^ recover 
the mapping of the terms into the categories (sections) merely 
by counting the number of times' that a term occurs in each of 
the sections, taking into account the fact that the sections 
have different overall numbers of records (and hence proba- 

^^^^ ^^^^ ^^^^^ occur in a section) , and looking 

for peaks in the distribution. When this is done for a typical 
term, using the 80 CA sections, the result is a plot such as ' 
that shown- in Figure 19. Terms that occurred mpstly in one. 
section, like Term A, are characterized by the subject of _^ 
that section. For instance, the term "estradiol", which is 
the name "of a hormone, occurred almost exclusively in' the 
section on "Hormone Pharmacology" (Figure 20). Hence, 
independent of any use of its dictionary definition, "estradiol" 
was iden^tificd as a hormone pharmacology Jype word. Other 
words like Term B, have a broader distribution but are still 
restricted to a limited range of sections, such as those 
relevant to organic chemistry, inorganic chemistry, etc. An 
example of this type of behavior is the term "fiber", which, 
as sho.wn in Figure 21, occurred mainly in the sections on 
"polymer Chemistry". Other terms, such as "acid". Figure 22,' 
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Figure 19.; The Relative Frequencies of Four Hypqthetical' Terms 
• • in Lach of the 80 CACon Sections 
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Figure' 20. Distribution^^f the Tern "ESTRADIOL" in CACon, 
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or* ''pressure" Lave distributions like teirms C or D on Figure 
19. The meaning ^of such flat distribution^' is that the terms 
are equally applicable to the concepts of 'each of the CAS 
Chemical Abstracts sections. This need not me^an that C or D 
terms are'not^good discriminating words. Rather, it just 
means that their discrimination value is very limited with 
respect to the term classes consisting of * CACon section, 
labels. For instance, a term related to temperature or 
pressure may be of conceptyial value for retrieval and may 
occur 'in only a small fraction of records. Still, if its 
distribution is flit, i.e. if it occurs' equally in all CACon 
sections, t1\en it 'cannot be assigned to a CACon section 
term class. . The major advantages of this form of <term 

■ ' . ' 

classification are that the term classes and tjheir heading-? 
are based on. intellectual judgement?. That is,' records (and 
the ^erms they contain) are assigned to sections by in'dexers 
according to their record meaning. That is, indexers assign 
records to sections according to the meaning of t-he section 
title and terms. Further examples of word" distr JLbutions are 
shown in Figures 22, 23 and 24. 

Examination of the 'dlsCributions of all the terms in 
two issues of CACon shows that most of the terms map easily 
into either a single section or .a small group of sections. 
Some terms, such as "absorption", map into two sections or 
giroups of sections, because they can have two separate • " 
•meanings, as the sense of physical absorption versus spectral 
absorption. ' 

To characterize the degree to whicftil the free text terms 

of CACon map into section or supersectioAs , th'e distribution 

such as those thown in Figures 19 to 24, v^a generated for 

each test Cerm. Then the fraction of normaVized occurrences 

of a single term that occurred in the peak /section of the 

distribution was .calculated according to \ 

■ - 
" f, = the number of occurrences of d ter> in its peak section 
the total number of-occurrences of the term 
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The term counts are normalised to account for the fact 
that different sect:^ons contain different numbers of records. 
Similarly, the fraction of normalized occurrences o£ a term 
that occurred in the section with the second greatest concen- 
tration of that term was calculated according to: 

f _ the number of occurrences of a-term in its second peak section 
^ che total number of occurrences of the term 

The fractions fi and fz have the following propertii's. 
If a term occurs only once in the record set, fi=^l and £2=0, 
That is, if a term appears only once, then it can appear in 
only one section and so it must map into one section per- 
fectly (fi=l) and into no other (f2=0). If a tednn occurs 
only twice, then fi+f2==l, since the term can occur in only 
two sections if it appears only twice- -In general, the- 
closer that fi is to 1, the better that a given term maps 
into a single category. Of course, aside from singular terms 
few terms approach fi=l. Moreover, if fi did equal 1 for a 
given term, that mapping would be of little value as a recall 
device (since any record containing that term could be ob- / 
tained by searching on the section name). However, it still 
retains great value as a precision device, as it may still 
be used to partition records within the retrieved set. For 
example, suppose ''estradiol'* occurred only in the section on 
"Hormone Pharmacology". Then, all "estradiol" records could 
be retrieved by searching on the section name rather than on 
•'estradiol". However, "estradiol", still separates records 
into two classes - with or without that term - and so it is 
still valuable for precision. In fact,, using the data for 
Figure 20, the term "estradiol" peaks in Section 2, with 16 
occurrences, and the second greatest peak occurs in Section 
13, with 4 occurrences. The total number of occurrences of 
this term is 30^ Hqnce, (except for the normalization), 

f, =^=.5-33 - • • 

f . = 3^ = '.,133 

fi+f2= .666 



So, for the term "estradiol", 66.6% of the unnormalizcd 
occurrences occurrences occur in two sections. Similai; data 
for all terms is presented T!n Figure 25 through 34. For these 
vcalculations , the _low frequency terms (less than 25 occurrences 
In two CA issues) were treated separately from the high frequency 
terms. The reason for this treatment is that low frequency 
terms may tend to occur in a small number of ;sections simply 
because they occur only a few times. 

The high resolution of the term iriap suggests* a method for 
overcoming the problem of selecting terms to feed back to the 
user that. was identified in the first experiment. The 
searcher has only to name a term class of interest (e.g. 
"Hormone Pharmacology") and only the terms that belong to 
that class (such as "estradiol") and are present in the 
retrieved set will be identified and sorted for feedback. 
This procedure xTOuld simultaneously focus attention on the 
key term^ distinguish between content-specific and content- 
nonspecific terms, and simulate the general mechanism by which 
context is specified in discourse. 

- The average value of fi for high frequency terms, 'from 
Figure 25 is about ^.55, which means that the average high 
frequency term has 55% of its occurrences in one section. 
Fi^J^fe 25 shows a similar plot fbr the second peaks of the 
high frequency terms. Since a second peak must necessarily, 
contain less than h^lf the occurrences of a given term, the 
curve falls to zero somewhat short of f2=50 (actually at 
f2=48). The average value of fa for high/frequency terms is, 
from the data of Figure 26, is about J2 5, so that about 68% 
(fi+f2) of high frequency term occurrences are accounted for 
by the first and second peaks. 

Figure- V and. 28 contain similar data for the low fre- 
quency ten .s expected, the very large component of low 
frequency te- .s that maps uniquely into a single section (fi=l) 
is composed almost entirely (over 95%) of terms that occur only 
once. Most of the high frequency terms that map uniquely into 
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one section are . indexing terms that are assigned by CAS to 
the records. 

^Figure 29 presents data for the values of fi for the 
^*high frequency terms. The dis1:f ibution is remarkably smooth 
and well behaved; and it shows that the concept of ATC is 
likely to work because so many terras have such lai:ge fractions 
of their occurrences in single sections. 'More than half of 
the high frequency terms each have more than half of their 
normalized occurrences in a single section. Since there 
ar-e 80 total sections, the average fraction of term occurrences 
that-wbuld be expected^ in a section of rhe basis of chance for 
a randomized distribution of terms (no significant cqrrelation 
of term occurrences) is only 0.013 (i.e. 1/80) J In contrast 
to the observation that most term occurrences are uncorrelated 
with each other,^^'^^ the correlation between terms and 
sections is very high. - » 

Examination of tme terms* that have low values of fi 
reveals that they ar^ the very general terms, such' as. ''theoi 
''review'^, ^'experiment" , "effect'', etc. These terms should ^lot 
map well, and the mapping^^te^H^ique provides a convenient 
method for isolating them. It is .these high frequency "terms 
which are not context specific that degrade the contribution* 
of the abstract field to the resolution of records in Experi- 
ment 2. The mapping experiment (4) provides an easy method by 
which these terms could be grouped int6 a separate category 
from the context specific terms, "if this were done, the reso- 
lution contribution of the abstract field should assume its 
expected dominant position among fields. Even discounting all 
the terms wi^^ fi~l, the remaining low frequency terms average 
fi=61 that t'he low frequency terms (even excluding terms 
that occur only once) map ver> well into just one section each. 
Also, low frequency terms average f2=25 so that, excluding 
terms that OQcur only once, about 86% of normalized low frequency 
- tenn occurrences are in only two sections per term'. < 
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Figures 29 and. 30 present the cumulative frequencies for 
the high and low frequency tar^j^/ V^hat is, suppose that a 
threshold were set (Fi), and'only terms with fi>FiWere mapped 
How, many terms would' be mapped for a given Fj? Figures^ 29 
and 30 give the' answer. For ^instance, *if Fi=0.3, then 72% 
of the high frequency terms and virtually all the low -fre- 
quency terms would be mapped. ^ . * * 

Note that this resultr is in harmony with the intuitive 
notion that the lov/er frequency terms are mote content 
__sEecific, for th^ogcjurrje^^^^ 1 ow..^requeucy.^ 

term are more concentrated into a single section than are 
the occurrences of the average high frequency term. 

• Figures 3^1 through 34 contain similar data for the 
distribution of terms over supersections . Since each super- 
section is composed of several s'ections, the fraction of 
occurrences in a given division, f j , must be ^reatfer or equal 
for supersections as opposed to sections. Remarkably, 94.7% 
of high frequency terms map into orie supersection with fi>.99 
'A similar, statement also holds true for the low frequency 
terms, distributed over supersections. Clelarly, the super- 
section division of terms is much less demanding' than the 
section division and denotes a second very valuable level to 
tne mapping hierarchy. \ , 

The vocabulary mapping experiments show that simple 
statistical sorting operations applied to manually indexed 
data base can yield a very useful hierarchical mapping of the 
terms Anto categories; -Tt-"now remains to be shown" that these 
categories prove useful for the IR tasks that have motivated 
their construction. In the spirit of' the prev?.ous discussion, 
the statistical intellectual term cl,asses offer the following 
method for -overcoming th? limitations of binary comparison.' 
For the example of "dog", "greyhound" and("b4an", the first 
two terms map into the "Mammalian Biochemi^y" sections of I 
CACon (CAOll). ■ "Bean" maps inCo the "Plant Biochemistry" 
section of CACon (CA017). As before., "maps" m6ans t^hat the 
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term has its greatest concentration in the given section. 
Now^ if each term is augmented by adding the class name to 
it, the following situation arises: 

Bean Bean-CA017 
Dog Dog-CAOll 
Greyhound Greyhound-CAOll . 

No matches One link between Dog and Greyhound 

at distance = 1 ~ = 0.67 
That is, "dog'* is linked to '^'glreyhound" at a distance inter- 
mediate betr^een identical match and no match. Augmented' 
identical terms still match at zero distance. 

The principle of augmented terms can be applied at ir.ore 
than on^ level. Thus, a term can be autmented* with the names, 
for instance, of the CACon subsection, section and supersection 
in which it occurs so: 

\ 

Term 1 • CACon Subsection 1 • CACon Section 'l ' CACon Supersection 1 
Term 2 • CACon Subsection 2 • CACon Section '2 • CACon Supersection 2 
If Term 1 is identical to Term^2, they are joined at 
distance zero. If Term 1 is not equal to Term 2, but they iftap 
into the same subsection (So CASub 1 = CASub 2. CASect 1 = CASect 
and CASuper 1 = CASuper 2) then Term 1 and Term are joii - i at 
distance = 1 - ^ = .4. Similarly, if the CASuper 's are equal, 
the connection is at distance = 1 - i'= 0.86. The progressive 
distances of the connections j.oins at different levels of map 
relatedness are in close correspondence with intuitive expec- ' 
tations Qf desired term behavior. Moreover, the simplicity of 
the procedures means that they can be performed inexpensively. 
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ANALYSIS ' 

The three critical parameters that characterize a clus- 
tering run are coverage, agglomeration and accuracy. By 
using a statistical model of the clustering process (assuming 
th^t term occurrences are largely uncorrelated) , and a simple 
measure of term distribution, it is possible to predict the 
cov^^^rage and the agglomeration ais a function of the cluster 
distance, 'the model also predicts which terms will be domir 
nant in forming the pattern and leads to^ recommendations for 
mpdification of the shape of the term frequency ' distribution 
to improve retrieval efficiency. The model doe^'not predict 
the accuracy of record assignment to clusters. Howeveir, 'one 
can readily use the model to calculate" the degree by which 
an experimentally determined set of assignments exceeds the 
chance level. By using experimentally determined clustering 
accuracy as a function of measures of the term distribution, 
estimates of the usefulness of cl„ , ^ering in new situations 
can be made. The excellence of the a'greement betj^een the 
model and the data supports ^the assumption of uncorrelated 
t^rm occurrences, "^in support of the literature^ ^ ' ^ 
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'STATISTICAL HODEL OF CLUSTERING C OVERAGE 

1 . All Term Frequencies Equal 

Suppose that in a collection of " Np records, there are J 
unique terms, each of which occurs with the same frequency, 
rt. (i.e. eadh of the J terms occurs in the same number of 
records). The cas^e of equifrequent terms is simple to test, • 
and' can readily be generalized to describe the case wherein * 
the terms each have their own frequencies (each term may 
occur, in a different number of records). Morepver, assume 
that each record ha's the same numbe.r^of terms, N^. This is 
a good assumption for the CACon data' base. Note that N-=?jJ_ 

Represent each record by a J- tuple. Let a 1 in the j th 
position correspond, to the presence of the jth term, and I'et a 
0 correspond to its absence. For each record, the corresponding 
J- tuple will have N of its positions fi.IIed with I's. To 
calculate the number of records that are clustered at a given f 
distance, one merely has to calculate the number of 'records • 
that share at .le.ast k terms with at least 1 other rec^ord, where " 
k is determined by the distance formula 

k 



J 



D = 1 



2!I^-k 



• So k = 2N.^(1~D)/(2~D) 

t 

Given any two recoros froin thi collection, the probabilit: 
that they will match, on at least one terTn..is easily calculated, 
Since all the terms have equal frequencie- . the probability 
that -any one term is present in a given record, is the same 
pnoblem as the probability of picking one specified ball in 
chances ^om an urn 7ith. J numbered balls. 

The probability that there 'is a' match on Lhe jth term is 
the product of the probabilities that the ; th term is ptesent 
i . each of the tw,o records . Let : 



er|c . sv 



p(jO = probability of a match on the j t h term 

P(j^ = (probability that the jth term is in Ri) (Probability 
the jth term is in R2 given that i't is in RO " . 

N, N.-l 

P(k) = probability that there are at; least k term matches 

between Ri and R2 ' ^ 

P(ex k)^ probability that there are exactly k term matches 
between. Ri and R2 

PXex. 0)= 1 - ((l-p(j))^ ' . • - 

That is the pr'obaUility that there are no term matches 
between two records is 1 minus the product of the 
probabilities that .there is no term match on any of 
' ■ the J terms . 

Ln[l-P(e>:0j = Ln|((l-p(j))'^J= J.ln(l-p(j))• 
forp(j)<<l, Ln(l;p(j)),; - p(-jj 
So Ln[l-P(.exo] - -J^j) . ' ' . 

1 - P(.ex 0)= expC-JpCjy) 

PffixC)= l-exp(-Jp(j)) 

So, the probability of at least one match is I minus 
the proba^biliuy of no matches, and 

X 

pa) = exp(-Jp(j)) . -, . 

P(l) = exp|"-J ■ Si . filil . . ■ 
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Case of Non-Equal Term Frequencies 



Ti 

T2 



Ri 



'Ti, 
T2 
T3 



R- 



As an example of partial record sets with terms of ' 
unequal frequency, consider records pairs (Rj + R2) and 
(R? + Ri,) : For (Ri and R2) there arci 4 possible terms 
(J = 4), all of equal frequency. Suppose = 1, Then 
there are 4 matches out of 16 possible combinations for a 
match probability of '^5 = 4 at a ^disfance of 1 - -ffj-rr = 

1 ' • ^ 

^"JTi -0- Suppose that R3 and Ri, are identical^ to Ri 

• " /I "> 

and R2, except that" the first two terms are identical,- 

(i e. the first t^tm has twice the frequency of any of the 

others): ^Thus, there are, in effect, 3 terms (j =.3), one 

of which has twice th^ frequency of the other two. From 

the diagram, there ^re 6 matches out of 16 possible com- 

bina*:ions for a match probability of |. So, it is 

clear that for cases of unequal frequency, each term 

contrilputQS t;o the matches approximately^ according to^the 

square of the term frequency. ^ 

When the derivation of P||^ is done for the c^se where' 
the terms are each allowed to haVe distinct frequencies, 
(See Appendix B) the result is found to obey a Poisson - 
distribution. 

K-1 l^.-L 
P(k) ^ I - y L e 



fot k>l and 



N. 

Wl<<l for ail 
F 



for Nj comparable to Nj., (which corresponds to t,he case v/here 
one- term occur.) in most records) , additiqnal factG^rs of L 
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occur in .the result. ^ In this expression, 

• L = the average number o? term matches (links) per 
record pair. ^ » v 

^ Np(N -1) 

Since the number of ^record pairs is , , - 

^ . z and. the 

number of term matches is" A, (N.-l) " S N.(N.-,1) 

2 ^ ^F/Nf-1> ' 

It is useful to note that the equation for P(k) depends 
only on the parameter L. Since the shape of the cluster- 
. pa^tcem depends on the number of links formed, one may ^sk 
which terms contribute most to the formation of a pattern. 
Clearly the single frequency (one appearance onlyf terms 
cannot- contribute much to-.a pattern since they cannot -produce 
a-link; I.t has been argued by others .that.sileh terms 
contribute to the pattern b> identifying dimensions' along 
which records are dif f erent ^ 5-. • That is true, but the 
experiments show that terms ^e so weakly semantically linked 
that singular terms only degrade the pattern, i.e. degrade 
the significance of the matches. ' * 

Higher frequency terms contribute. progressively more 
to a- Fiutern. A term with a record frequency of N. contributes 
-a number of' links fN.l N^. (N*. -1") 

' ■ L2J ^ 

For = 1 (singluar terms) it is zero. ■For.N.>a, as'expected, 
it increases as H j . Because there are very many more low 
frequency- terms control the overall eluster pattern for 'a g-'.ven 
case. Figures' 37 and 38 indicate that sometimes even a sin5le 
high frequency term can overbalance the lln' g power of all 
the low frequency terms. ^ This work' suggest .hat it is not 
sufficient to report -N^, N^, J and ^ when documenting clustering 
experiments. It is also desirable to report "the average nuuber 
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of links per record pair , (L) 



If there are any terms in the file for which Np,~ Np , these , 
should be repotted too (See Appendix 3). It is for this 
reason that typical distributions, rather than average 
^di'Stributions* are plotted in figures 35 and 36, i.e. since • 



N? 
J 



N?, to calculate L v>n the, basis 



of average term frequencies would underestimate the signifi- 
cance of the high frequency terms. 
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I^OSnber df RetoVds Clxiste^ed - Multiple Links and ARRlomeration 
■ «ar " . ^ ^ ; . 

^ Nw, the. number, of records clustered at a given distance 
. ^D|,"Nh;'= (Prob of at least k links between R\ **ard R.) • (Number . \ 
of -k. an^ R," ^airs) v (Number of records -x: lust ered per link) 
. whefe k linkrr assure ^kDi ' * • ' " . 

•v. ... - . ( 

^^-f(*i.,M) j^apress^es th-e- fact that when new links are 'formed they 



.may either' inwlsTe. previous linked/ records or not, as shovm 
oh* Fi^ure^ 39. * figure 40'^express^ f(L,M);. calculated expli- 
*ci.t>iy for M=l00. f^ote that; for'L.^G , -^iL^2 because every 

new linj^ is.,a 'type 1 link and binds tw6 p.reviousiy u^ . .vnd 

records-.^.-/ ' am 
:L --N AN 

' ^ * ^^'"^ iT" '^^ ITT ^ t^ecause most new links are type 

Z links, which bind ojie previously 'abound record to other previ- 
.-x>usly bound records. ^ ' • ' ^ ^ 

N ./ AN . - 

/"^sf-i. si^-o .• :■ ■ . . 

because liew linlcs occur primarily as type 3. whic^ on;Ly bind 
reviously bouad groups together. 

■f- 

Old Old 
•type 3 

, Figi're "35<^>5^^ pes of Ways that New Links Can Occur 

j-t has he shown by derivation and explicit- calctiliation 
thac^-it Is'^oughly true that ^ \ " 

^1 N^..N^(i-«xp(-|L)) ■; ^ 
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Figure 40. Link Redundancy Factor vs Ntimber of Records Clustered 
9^. for a 100 Record File 
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V The following graph shows the ,data of Figure 14, The- 
line is that calculated using the above values. The. curve 
matches the average of the r^levant/nonrelevant experimentaL 
coverage within one standard .deviation of the mean. 

The coverage model was also^ tested on the data of 
Experiment 2. A^' shown in Figure 42,. it f^ts.the data^.well 
for various conditions. • ' t,. ' 
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STATISTICAL MODEL, OF AGGLOMERATION 

• •■ \ • . 

The average cluster size depends ,6n the number of type 

«1, type 2 and type 3 joins (ni,n2^3/) respectively. ^(3ee ^ 

Figure 35). 'The numb er*^ of separate clusters is appro:5imatei7^ 

nWna, since an ni join creates .a* 'cluster and for N <<N , an 

na'type'join usually destroys priel An na join neither' 

creates nor destroys a separate clusterv, but rather it just 

joins a previously un jo ine^^ record., to an. exfstant- cluster.' 

Hence, the average number/of records per ^cluster , 1^^, Is , ' 

'ven .approximately by:/ ' . , , ^ 

; / ' ' * ' / . 

. ' = ^ yfbr n3<ni 

^ ^ ni-n3/' 



Where: ni = Nupiter type 1 links 
. .n2 =.I^simber of type 2 linkc 
1^3 =/Number of type 3 links 



L/.= ni + n2^+ na 
So: ' ;{2 = 2L - 2ri3 - N, 



/ 



ni N -L + na^ 
c • 



So/ -N: ^ 



/ 



• N_ .N_-N 
/W: L - - / In -V-^ 

■So: . N, ^. ' ^ 



// * A ■ _^ Np r N_-N. T 



/ / TO— in 



' ^ c 

For: N^>N 
F c 



This eqjjation is plotted on Figure .43 for = 100. Agglomera- 
tion becomes appreciable when ^^c ■ ^ ' ' 

If- .6' (i.e. 60% of the file 

is joined at least once). \ ^ 



^ . "101 ■ 



Using the, data of Figure 36 to telate to D and the above 
-tequab^n t.o relate to N^^ results in Figure 44, onVhich" 
is superimposed the data of Figure 16. The above equation 
fits the data very well up to ^J^^Zj75. ' A^ove that level,' 
the- number of type joins that do^nojt unite clu'sters becomes 
appreciable, and a more exact treatment is required (Based on 
resolving the two possible kinds of. type 3 joins).. The, simple 
^equa^ion, however, is 'sufficiently a^icurate to serye- as a - 
.guide to system desl-gh."" . > ' • 
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STATISTICAL MODEL 0? THE AbeURACY OF CLUSTERING RECORD ASSIGNMENT 



The procedure used in evaluating a cluster, for experiments 
2 and 3, wliereln each record belongs t;o 6ne off two classes 
• (relevant vs. ,non- relevant or CACon Section X vs. CACon Section 

. Y) is to tPtal the number 6f records of each type .within a 
cluster, and assign^ the-clus^Tsr to whichever class 'has a 
majority., For instance, if a cluster contained 10 records, 

. of which ,^ were r^elevaht and 3 were, non-relevant , the. cluster 
would be designated relevant, 7 as^slgnments ^ould be counted 

•aS| correct, and 3 would be cfeuntea as errors . , Hdweveir , ^ it 
is not correct to deduce, froid this aata^ that the^ accuracy ' 
.of clustering record^ assignment is 70%/ Rather, the, assign- 
ment 'perf ormapce must be CQjnpared with the f;requenc7 with 
'which correct, assignments would be made by ch'^nce alone. ^ 
^Fpf j:he case of a 10-record cluster, no 'more than incor- 

. rect assignments can be -made. In other words, even if records 
were ai^signed to clusters on the basis of chance, because 
clusters are labeled as being, type A or type B based oh their 
majority constituents, no more tl^^. 5 incorrect assignments ^ 
Qould be made to 'a 10-record -cluster. A more detailed ex^i- 
^ nation of tile Statistics shows -that the '^average chance level 
is somewhat greater than the minimum. Recall that for *the 
experiments designed, there were always equal numbers of.; the 

_ two kinds of records in the set to be clusrered,. so j:hat the, 
probability th^t a given record is either one type or another 

, IS .5. < , • 

Fox a 2-record cluster, there are 4 possible- combinations 
of records ^ . ' , * 

Combinations Score 
. • . +r 2^ . • 

+^ ' 1 • 

" , ^ • *-+ * 1 • 



4 - Total , 6 = Total 'Score 

Combinations 



^ ' 8'5 

mc • .. 105 



\ 



Since each combinatibn is equiprobable (approximately), the- 
average^ scare attained by chance for a two-recor<i cluster is 
1.5 f(i,e. 6'5'4).' Similarly, for a 3-record cluster, there are 
8 combin^itions:- * ^ . ' 




Combinations 

$ 




^"^^^ I I 1 ^ 
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' 8 Total Combinations 18 = Total Score 
For this case, the average score attained by chance alone is 
18 



8 



= 2.25 



Calculating -the chance levels for p.rogressively 'Larger 
clusters leads to the curve shown in Figure 45 1 As is shown 
on that figure, the relationship between the average score 
attained by chance and the cluster si2e is approximately 
line'ar, and may be -estimated reasonably well by the 'equation: 



■ . 625N +. 
c 



.30 for N i2 
c. 



This equation may be, used to .calculate the Extent to which 

a given set of^clustefs fexceed the cjiance. level "in t^e -accuracy 

of' their record assignments. .)'*'.• ' 

The score attained by a cluster run is calculated as ' 
the fractiop of total assignments that are correct ,. above the 
chance level (S) . At any given cluster distance, the number 
of records clustered (N^) and the number of clusters (N) are 
tabulated, -so that the average number of records, per cluster - 
(N^) is: ' • " 



So; thfi total .nvimber' of correct assignments, by chance alone ^ 
(Nj^c") is tihe ,nvimber of correct, assignmen^ts* per cluster? times • 
the number of ^clusters - - ' \- 

'Njj^ total cc3rj:,^ct cluster as§iginfeents , .t^e score is 
. given by 




S has the -properties' that S=0 if the assignments ar^ correct 
.1 ^ . .<■•,'»• 



>^ ' ' . onl^at the'-chancev lev^.^ 

^ ^==^1 if thje assignments aire ail' correct 

' ' . * ■•. ' • ' 

l>S>0-if ■rI^>Nt^>Nj^^"..and';;S is lYnear with Mj^ . \. 'V-v'*' 

Applying this, formula to the data on Figlire 13 leads to ■ 
Figure 46. _ -It is ^eslear from thi.^ figure thafE the accur^^^ith 
which sin:pl,e clustering makes recoj^d assignment^ tb clusters is 
very substantial- (aboue 807.)* for "cluster distarfcfes^es^ than ".5, 
but that at larger distances it; rapidly falls off to junaccepCi- 
ay.y low values." This is not . surprising,. If two records Ijave 
'50% o'r more of their terms- in 'common, it is- not surprising that 
they should be grouped together. Also, if two records have 
only about 20% .o^ .their terms in common, it is njSt .Surprising ' 
that grouping is little better- than chance. t 
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, , At a distancfe of .5, only about 13% of records are 
clustered and the average, cluster size is only abo.ui 2.7 
•records per cluster, so that the number of user decisions 
have only been reduced from 100 to about 94^ (±.^, 
^c 

Np - (N^-l)»jr- = the number of user decisions required). 

"This petfonnance is not significantly 'beneficial to irhe user. 
This ana^lysis reemphasizes the need to incorporate semantic 
infonftation into the system, in order to incr&ase S at larger 
values of distance, 'where the^eduction in the number of 
required us^r decisions is more significant. ' 

Examination of individual runs shows that the primary 

reason for incorrect gt^upings is the failure of semantLcally 

ft j 
related but non- identical strings, such as greyhound and dog, 

to match. This is a' problem that cannot be solved by a change 
in the choice of clustering distance measure, because changing 
the measure cannot recapture the. semantically buried information, 
Rather, a means is needed to record the conceptual relatedness • 
of terms. ATC is. an approach to this end using statistically 
constructed intellectual term classes. Because the term 
classes always map terms into groups with larger values of 
Nj, th.e mapping is subject to .the criticism that it sacrifices 
■percision for recall. That is, Salton has conjectured that i-t 
is the intermediate frequency terms that are the most' important 
-for information retrieval^". The very low frequency terms, 
it is argued, cannot. be very important because they cannot 
participate in many matches. Also, the very high frequency 
terms cannot be very significant because they lack 9pe<?if icity , 
i.e. they match so often that the -information value of a mi-tch 
is small. Accordingly , " he recommended that very low frequency 
terms be grouped Jsito intermediate frequency classes, and very 
high frequency "terms be divided up into intermediate frequency 
term phrases. These suggestions seem unassailable in^ the 
context on one-step searching. Yet, in the context of multi- 
step searching, it seems preferable to use the structured 
vocabulary methods , described in Experiment' 4. Representing - 
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no 



terms within such an hierarchy allows for the matching to 
be performed within the limitation of a given range of concepts 
Xthe idea of SBC), and to match sttings that are not identical 
with a match value less than unity, and to perform the matches 
at selected levels of generality. The process of adding to a 
term the names of the categories in which it is found is^ to 
carry' with the term the context of its use. Williams found 
this kind of information useful in directing a user query to 
an appropriate data base^^ It is just this kind of informa- 
tion that is used implicitly in dialog to break the ambiguity 
of term definition. Thus whereas "absorption" has two dis- 
tinct definitions (at least) they may be disambiguated by 
noting that one is in the spectral sense and one is in the 
physical sense. The use of a formalism in which the spe- 
cific term mappings ar,e associated with a term occurrence 
suggests a naeural interface with artificial intelligence 

' ***** 

processing tasks. Using techniques.^ perhaps terms can..^ 
be disambiguated by consideration of , the contexts in which 
they occur . ^ Similarly, the occu rrences o f the labeL ejl term 
suggests that the identification of the contexts would be 
made easier as x<?ell, perhaps through local consensus. 

The effects of vocabulary mapping can be evaluated in 
terms of the statistical—G^ltis^t-ering model- Every word in 
the language is a precise instrument, and any time virtually 
any term is replaced with aribt her, meaning is changed. Any 
time^that meaning is degraded, the accuracy with which records 
can be grouped is depressed. Of. course, if terms are replaced 
yy more general terms, EN? is increased and the probability of 
match is increased, so. that coverage and agglomeration are in- 
creased. The experiments performed suggest that for accuracy 
to be sufficient, coverage must approach 100% at a distance of 
less than about .5. Convenience would suggest that average 
cluster size should be about 5i^at that distance as well. The 
statistical model predicts that these i:pnditions would 
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require - 13,500 for affile of 100 records. The^aptual' 

.J 

value of in the experiments is about 5100 • 



- -Rough caicu-lWions^ show that ATC can achieve the factor 
'of about. 3 that is required to raise ]^N? to the projected 
feasit)le' range- By increasing the number of links between 
records, ATC can be projected to achreve .resolution of 
relevant and non-relevant records to a degree* that' is' 
useful to a user. However, this projection should be 
regarded only as a motivation for further work, and not as 
a guarantee* of success. ' ^ / ^ ^ 
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^FRDCESSING COST 

The -costs involved 'in Itpp lying simple clus taring. tt) about, 

100 bibliogra|5Tiic records from either CACon or Ei COMPENDEX 
include identifying the;, terms, applying a stop list, utilizing 
controls and, finally, clust'eriiig. In experimental runs, on an 
IBM 370/158, these steps consume about/20 cpu seconds for the - 
term preparation and ^20 ^pu f<5r' thfe ci^ust"ering. In production 
runs, the computation time would be considerably less. Much 
of the term identif icatibn process could be saved by pre- 
processing the records (i.e. sttxring stems and stop-listed 
terms, perhaps in a canonical form). The clustering time 
could also be greatly r&dnce^: 'The experimental runs gave - 
much m^re detail than woulS be required by a user. Perhaps 
15 cpu seconds would be. a reasonable estimate, for 100 records 

and about 60 cpu for 1,000 -records. J 

■ ' ' • ' . 

The ATC term njapping requires about 300 cpu sejc ends for v 
.two issues of CACon. This is the cost for associating a term 
with a^ubsectibn, section and supersection. Several hundred 
more seconds ar^e required to restructtire the data base-. to put 
it- into a form to take advantage of mapping. ^ 

The SB'C clustering should cost less than the simfhJLe 
clustering because fewer terms, per record are accepted' by the 
content focusing mechanism. Howetfer, firm cost estimates are 
not available yet for SBC. 

The ATC term mapping, structuring /and labeling* operations 
are done only once' on a data base and are then available, for 
all searches. .In. essence, global information is processed- . . 
opce, saving each separate user from repeating the same in- 
tellectual operations. 
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5 . ^DISCUSSION 



How is an IR system! to be made efficient? For string 
processing programs, ^thej historical first step was to save 
on the number of string Icompares ..required during single re- 
trieval. Inverted, files' handle ^that phase very well by 
sorting the file into a structure such that the anticipated 

, question, "Wher<^ does s tiring xxxx' occur?" is answered f.pr 
alT strings before any ^earches' are done. This saves eacp^ 

>ser the costvof doing" that sort separately. On a somewliat 
more sophisticated level, ATC similarly saves each user/:^rom 
ari^lyzipg^ theJ%^ntext o|f each term by using global sta,fc''jis- 
tical information to relate all the implicit <:ontext , def ini- 
tions before amr searches aire done. That is, just is string 
processing progfvams save on .comparisons 'by comparing /pnly. 
those strings forvwhich a match is possible (based ofl a 
crude, first approximation such as LCB^^y g^jj^^^^^..^ p^V^^ggj-^g 
programs Should save on compares by using a crude first 
approximation to meaning (such as ATC) , / 

When one projects the structure and capabilijiies of the. 
IR systems of the future, one is inevitably drawi^' "to consider 
the automation of semantic and cognitive processes. (i.e. 
the functions performed ty ah ideal librarian.) / In this 
regard,. one Is led to ask, "What is the future fole of current 
statistical strijig processing procedures in fufcure systems 
that will be doing semantic processing?" It is ' tempting to 
think that. the future IR system would be a "world brain" in 
'which statistical processes had no place, i.e. where new' 
information was folded into ah -existing^ knowledge bank by a 
, process analagous to "understarlding" . In such a circumstance, 
one might assume that retrieval would be very fast^, analagous , 
■to the human power of ^abstraction of concepts , ^ However , there 
\re two problems with this point of view. .First, even for 
humans', recall is statistically ba^ed. f Frequently used infor- 
mati^^ is easily retrieved in the human mind while infrequently 
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used information is often remembered only with great dilfficulty 
Mbreover, such performance is reasonable. . First, when memory 
space is finite, and response time is important, it makes 
sense to put the highest priority records in the most acces- 
sible places'. The second ^problem is that the process of 
understanding generally i^eans developing the capacity to 
answer a given class of problems by preprocessing the data. 
For example, if I'm told that "John is in Te?cas'\. I ca^n 
easily answer the question "Where is John?" However, there, 
are many other questiojis sych as "Why -is John in Texas?", • ^ 
that are not easily anticipated nor are they easily handled 
hfy standard (canonical) forms. ' Such questions may require 
inference and .the use of implicit information. T?he point^i^s 
that the large number of questions " that may be asked about 
text is large, perhaps infinite, and no system can be expected* 
to have answfered all" of them /on a prep/:oc^ssor basis. Some' 
large classes bf questions may.be dnawerable on .a preprocessor 
basis (like" "I^ere IS John?"/), but many of the unantsLcipatible 
ques.tions will require, run plme analysis of records. To ,be 
efficient, it seems that th^e two kinds of questions * (high 
frequency anticipatible or/ low frequency unanticipatibl^) ^ • 
should lAse memory in diff6(rent ways. The ATC and sequential" 
search formalism has an obvious extension' that seems to 
accomodate these two needs. It consists of the representation 
of each term by an -n- tuple in which each field corresponds 
*to an attribute and each entry corresponds to a value.' For 
a multi-step system based on such a representation, the ^Boolean 
search component would access a "limited range Af, fields, inter- 
mediate processing would have access to more f itelds , and seman- 
tic processing would have access to all field?.! Such represen- 
tation has not been the f ocus . 0^., recent -AX research activity 
because of the apparent storage economies and the other 
successes achieved by semantic nets and linked lists. In the 
use'of these methods, every attribute is -a node. What t^s 
suggested hare is that: the nodes in a semantic net need not be 
bare character strings. Rather, they may be n-tuplffs. Then 
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the semantic net becomes a network between n- tuples. That is, * 
one ^ of the most serious problems in latural language AI is 
the prioritization of computer processing tasks. Processings 
demons are one attempt^ ^. ^Perhaps the high- frequency memot'y 
access' needs would be b^'st met by explicit n- tuple repres<in- 
tation while the low frequency nee^s would be met by pointers 
and semantic nat relations. ^ 

Reaction time experiments^^ suggest that human memory 
works on a bucket principle such that weak relations idetitify 
the bucket in which the words that -are candidates for a given 
usage are stored. Intellect is then ;s-equired to examine the, 
contents of a given bucket arid to' select the, appropriate word-. 
It i's interesting to note that if the n-tuple representation 
of terms were used as the .bucket forming mechanism, and if • ' 
each entry in the- ri- tuple were binary, fbr/.n'=20, there would 
be enough possibilities to disambi'gua^re-' iO'* words. The 20 p . 
bit strings would allow classes of .wofds with similar meanings 
*to be-retrieved directly through their similar bit strings. 
That is, the 20 b'it strings could provide a fast bucket 
retrieval mechani^sm for the content? addressibility of* terms^ 
It may ..be co-incidental, but in the gaijle of 20 questions, . 
20 binary responses to a more-or-less standard collection. of 
questions is sufficient to- -disambiguate (guess) the selected 
thing (word) from a collection of possibilities of the order • " 
of 10*.' \ - _ . ■ 

■The>^ overall point is that multi-step processing' of records 
consisting^ of terms., each of which is represented by augmented 
fields (n-kuples), some of which are statistical in origin and 
some of whi^h are semantic, seems'to suggest a system' design 
that can accomodate. levels of processing from' simple record 
retrieval to\deeailed AI.- This, work has demonstrated the vaiue 
of multi-step"^^ processing at the .statistical end. of the spectirum. 
wherein practical application to traditional Ik problems may 
be iiraninent involving user of the ATC pr related methods. More- 



over, it is'. suggested that application and interfacing of these 
methods with^ those in the realm of semantic information, 
prdcessing seems warranted, to tackle the IR problems of the 
future. • • . 
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mSt of all symbols .used 

D(a,b) = The distance between a and b as specified 
by a given measure. The. disl:ance may also 
be called "D" , x^hen a and b (or .their 
equivalents) ajre specified in another manner • 

f(L,N„) = The link r'edundauce factor, ^= the number of 
rep'ords clustered by ,L links fot* affile of 

: , * : * records • , , 

J • ^ = The 'nun^r of unique terms In a file^ 
L = The nraiber of links,* 

N ' = The number of. clusters in a file.- 

N(Rjj^,Rj) =^ ..The number of pairs of Records in a file- 

i • • • ^ * 

N-^(x) ' = • The number of records in a cluster;- 

' ; ^ ^ , ' ' ' ^ . • ^' 

Xhe.'number of records clustered, 

Nj. =' • The number of records "in a file, 

' ^ ' " ' " " * <j • 

N. H. • *The number of records in affile in which the 

^ , jth term (l<j<J> occurs.--^ 

Nj^ , = The number of records clustered' correctly , 

N.J,; = The nj^mber of t^rms in a record, 

* > • > 

P(k) * = The probabMity that two' records have at ^east 

k-terms in common (i,e. k term matches) . 

= the i'th record in a tile (l<i<:N«) • 

= ^ Accuracy of .clustering record .assigriments , 
allowing fo£ statistics ^af c^^ 
Tj = /the jth term^ina file (l<j<J). ^ 

accuracy - the fraction of clustered records that are. 
assigned to clusters correctly. 

agglomeration = the average "number of- records per cluster at a 
■ . ■ given distance. - . .' 

^"^^ = Automatic Term Classification 
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coverage = 

document - 
field 

precision f 
recall = 
record = 



^ni 

Ha 

fi 

i 



the number of records' in affile that are 
•clustered at a given cluster distance 

a publication or piece of one. 



a subdivis-ion of a xeoord. ire. author field, 
title fiefd, etc* 

the fraction of records retrieved that: are * 
relevant. 

the fraction of relfevant records in ^a data 
base 'that are retrieved, 

the represei:^tatioa of. a document in a data base, 
usually consisting of author, title', CODEN, and 
source fields. 

the numb'er pf tjrpe 1 reccr<l ^inks (i.e, 
number of new. links betwe.en previous unlinked 
records. ' ' * / • , 

'fhe number of typ^ 2 record 'links- (i.e. the 
niunber of new links betx^?een previously unlinked 
records and linked record's. 

. the n>imber-of type 3 record links (I.e. the. 
number of new links betvjeen previous linked 
re.cords. ^ * . 

the largest fraction of normalized tejnn occur- 
renceS', for, a single term^; in any CACon -division 
(subsection, section or ^upersec^tion) . 
the second largest fraction of normalized terpi 
occurrences, for a single term, in any CAGon 

* division (subsection, ^section or supersection) . 
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Simple Clustering = ^ clustering o cords without any 

preprocess^mg of the terms that they, 
contain, * V • 
SBC f Si^et Based Clustering, A clustering 

" " . . "* technique using term classes derived 

' from statistical preproc<?ssing to 
' * acco5nplish\jthree functions degrees^ 
of term mi^hes, term disambiguation,, 
'and- restriction of scope "of attention, 
^ = ^ the ^average number of- links^ per record 

' - /. pairv. / ' ^ 
]^(^) = • probability of at least k -term matches 

^ • , between, two -given records. 
P(j) - ~ =. • probability of a match- on. the jth^ term 

' •\ for txyo giv^n records, ' 

t^^^'^ ^) = , probabitlity of. exact ly -k term matches 

between Itwo given rec(^ds. 
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•TERM MATCHING EQUATIONS 



P(k) 

P(0) 

.P(j) . , 
>(ex k) 

'p(not j) 
L 



The number of records in- the file. 

The number of records with term i , N - <N„ 

Ji- F 

Probability of at least k term matches between 
t,wo records. 

Probability of"' no term matches between t^o records 
Probability of a match on the j."'th term. - ' 
Probability of exactly k term matches between 
two records . ^ 

Probability of no term match on the j'th term. 
Total number of links in the. file = 



N. (N.-l) 



N. 
2 

2 



/ 
/. 



= the humbe.r of pairs 



of " Identical records. 

The average ntimber o£ links per record pair 



P(j) 



P(j) 



"%%--l) 



total number of record pairs 

(probability jth term is in Ri) ' (probability 
that jth term is in R2 given that it is in Ri> 

' "1 

Np • Np-l • ■ . / 



P(ex'O) 



i JJp(not j) =17(1*'/ 



J 



j = l 



N. 

J. 



N.-l 
Np-l) 



■ , V"" V"" N, N.-l 

lnP(ex 0) = > InXno't j) = ^ 1^(1-^^- . j^I-y 

j=i • . j=i 



/ 
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for .N.<<Ntj 



J, 



P(ex 0)= exp 




N. • • ' 

Usually . is so small that it is a good approkimation co take 

' "F . ■ - 

P(exO)= exp-L 

I- ■ . • . 

If NjlNp for only- one j, .(denoted ^0'.'), then 

P'(ex 0)= exp-(L+ fg) for foE ' . ., 

^. , .. . • ■ ' ' . ' 

When 'all jj^<<l ^ . • 



P(l) = 1-P(ex 0)= 1 - exp-r 
P(2) = P(l) --P(ex 1) 



P(exl) = p(l) . p(not 2) • p(not. 3) 
+ p(not 1) • p(2).« p(not 3) 



p(not J-i) • p(not J.) 
p(not J-1) p(not J) 



•' p (not 1) • p(not 2) 
+ p(not 1) • p(not 2) 



p(not3)'« p(J-l) • p(not J) 
p(fiot- J-1) • p(J) 
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P(ex 1) = 2 llnot 2) • P^^°^ ^) • P^^°^ 2) ••• p(not J) 



= 1(0) t 



j=l 
J 



P(0) 




p(not j) 



i Np(Np-l) 



^<o) s . ^^^^ 

: P(0) •• L 



N.(N.-1),7 

1 _ J ]_ J 



L 



%(Np-l) 



L •■ P(0) 



Go P(2) = - e"^ • L 



and in general, for all Nj<<N^ 



:(k) =-1-2 



*L ^e ^ 



1^=0 



k>l 
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The software developed for this project is based' largely 
on programs previously developed at IITRI , including .the. file 
inversion software and several clustering programs , In order 
to conduct the'experiment^, software modifications were made 
and a few special purpose programs were written. These pro-r 
grams are briefly described below. . * ' 

- Standard Computer Search Center (CSC) file inversion 
software extracts terms from sp-ecifiod fields of each record 
■(usually title and keyword fields)^in a~file and associates 
with each one the number of the record (posting) in which it" 
occurs.- Small modifications of this procedure allowed 
different identification to be associated with each term 
occurrence. The most uffeful choice. was the CACon Section-- 
Subsection Number. The resiilt of the INVERT program is a 
file where each record consists of a term of up to 20 charac- 
ters followed Jjy a 6-character CACon. Section and Subsection 
number. ' 

This file is sorted on the term string within each block 
of entfries for a single .term. The^ entries are sorted on the 
Section/Subsection^ number field. This .procedure' places all • 
'occurrences of a^^ven term together, and orders the occur- 
rences according to| Section/ Subsection numbers. 

After the sort is completed-, multiple occurrences of 
any term in any Subsection will be , stored consecutively. Next 
the multiple record occurrences of each term are counted and' 
deleted. Then a new record is created which consists of the. 
term string followed by a list of all of the'postings for that 
-term. The CSC SQUEEZ program was modified to accomplish these 
ends. SQUEEZ creates, for each term, one. pr more varying length 
blocks each containing up. to 100 separate posting locations. 
'Each of these posting locations can accomodate all of the term 
postings within a given category (CACon divisions). That is, 
•if a term occurred in up to 100 separate Subsections, theii 
only one record would be needed; if between lOL-and 200, then 
2, records , etc-. Each block of up to 100 s^eparate posting 



12 1 



locations contains the nxamber of postings in that block, the 
fiist 20 characters of the term, the number of blocks created 
for the term so far, and the string of pairs consisting of the' 
Section and Subsection numbers and the frequency "of occurrence 
Within that section. _ This file format was ^chosen to facilitate 
statistical calculations,, of ^erm co'rrelations with CACon , 
divisions. (Supers ections ». Sections and Subsections). 

• ' The- first step i.n analyzing the inverted file is to nor-^* 
■malize the term frequencies for each section, to allow for the 
variance in t^Jie size of the sections. This normalization was 
based on the number of terms occurring in each section. ^ Inputs 
to NORM (the program that performs the normalization) total ' 
term frequencies per section alid the file created as a result^ 
of the SQUEEZ program. A noT^alizing factor is calculated by 
dividing the section, with the most terms by the niamber of 
terms in each section. A table'of these normalizing factors 
is created. The file is read thrp,ugh term by term multiply- 
ing the frequency /of the Se\tioh by t> ^ appropriate normaliz- 
ing factor in the table." The results are written into a new • 
file using the same structure t} ay were read from. 

'The file created by SINORM is used in. the second' step 
(S2SEC) to find the sections ./here the first and second pe^ks 
exist for a given term. For each term, the string CACon Section 
number and corresponding normalized frequencies are read (the 
subsection data are combined into section groups) -and the 
sections v/ith the highest frequencies are identified "and ^ 
printed out. Four 100 position arrays'" are declared to keep 
track of where these peaks have occurred. Each term increments 
a position in one array for 'the first peak and another array 
for the second peak. There are 2 separate arrays declared for 
high frequency (more thdn ?.5' normalized"- occurrences) and low 
frequency (less than 25 normalized occurrences)' "terms . These' 
arrays ar^ printed out at the end of the run. 
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Slight modifications were made to the second stepj-in order 
to obtain information on the first and second peaks within CACon 
Supersections. In.S2SUPER, the normalized section and sub- • 
sections are cpmbined into supersecl^ion groupings and the peaks 
are printed out as before. The four arrays are similarly in- 
cremented" and printed out. 

In order to exapiine these peaks for subsections, the file 
must be re-normalized based on term frequencies of the sub- 
jections. The file created by SQUEEZ is iis^d as input to- 
SI^^0RM2 which, along with S2SUB, produces output similar to 
the other versions of steps 1 and 2. Figure'C-1 summarized 
. this entire' procedure . . > 

the user relevance experiments and the programs used for ^. 
it, required the setting up of files with certain types of 
evaluated records. Th'e record numbers of the citation satis- 
fying a users prbf ile. as well/or whether it was denoted as 
relevant or nonreleyant isy the user was keypunched. With this 
infprmation the standard utility, program, 'SELECT, o6uld re- 
trieve these records from tapes maintained at IITRI containing 
the entire citations. These tapes of citations are, organized 
by Volume and record number and. contain records in a standard 
internal format. The citation numbers. of interest and the file 
of complete records for a given volume, serve as input to 
SELECT. These two inputs are sorted into record number order 
so that-these files may readily be compared for matches. When 
two recbrd number match, the corresponding citation is written 
out to a new file. Appropriate selectioff-of the input citatjon 
numbers results in the creation of a file of 50 relevant and 
5Q non-relevant complete citations for "a given user, the file 
created by SELECT is next processed by EXTRACT. EXTRACT 
organizes the term lists into a form convenient for clustering. 
Next, the subroutine TERMER is called. TERMER has 3 relevant 
parameters: a pointer to the citation, the fields to be analysed 
(CODEN, title-, etc,.), and a- character string of run time para- 
meters -that specify options such as inclusion of single occur- 
r-ence terms in the distance measure and output format.. 
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.,Terms are extracted by the subroutine TERMER. Under the 
directfion of EXTRACT, TERMER creates 4 set of lisfs of all 
terms fouhd, their numbers of occurrences and the Ipcations 

of those occurrences. \ - > * , 

• • • V 

_.The file created by EXTRACT is\sed by the program CLUSTER 
First the Document-Term arraCy is read and stored in a reduced 

, form. Calculations are performed for term distances. The 
resulting cluster analysis is printed out as a function of 
the distance value in dendograms and other data summary formats 
Flowcharts for the interaction of these procedures* are shown 
on the following pages. Computer listings for the major ^ 

. procedures follow subsequently. 
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Figure Cl . Processing Flow for Experiment 4 
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Figure C2 . Processing Flow for Experiment 3 
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• • • • rvxoA^-r ^AST UPDATt: 750103 «/ ' 0000 1 # 

EXTRACT J p^oC (RTP) OPTIONS(MA'N) I 00002 . 

/♦ THIS. PLAl^ PROGRAM EXTRACTS TEHMS fjR.^M CITATIONS AND DROPS 00003 * : 

SiMGULAl TERMS". INPUT IS P.L.^. FO MAT RECORDS OR OPTIONALLY A • 00004 
HIT-FILE. OUTPUT IS TERM LISTS FOR DOCUMENTS A DICITIONARY. OOOO'S 

THIS EXTRACT DROPS ALL SINGULAR TER'S ^ ASSIGNS A ^EftO JN. THE TERM 00006 - 

LIST ' . -*/ •,• 00007- 

DECLARE , ' • ^ . - • 00OU6 

NMAX FI<XED BIN STATIC* /♦ NUMBER OF RECORDS TO BE «EAD «/ ■ 00009 

< ALL BIT(l) ALlbNtD STATI-, ' OOOlO " ' 

^ 1 HIT.RECt . • , • 00011 

2 PROFNUM CHAR (10>» . 00012 . 

2-='HIT_WT FIXED DEC (!:)♦ ' . 00'0l3 

2 ;ABSN0 CHAR t}l)» .00014 • 

•a-.'SORT^F'LD CHAR (45) ♦ 00015 
2"HK^ST CHAR (79) ♦ 00016*'' 
1 CITJ^MBASED (P)» » 00017 

-2 ONCTXHAR (11) , • 00018 " ' 

2 REFNO CHAR (m* . ' 00019 

, 2 PAD CHAR (l), 00020 

2 DAT FIXED BlN» . ' . ^00021 
2- LOD FIXED BIN, . 00022 
2 LOP FIXED BIN, - 00023 

2 DIR (l) , 00024 
3 TYPE CHAR (4), ... 00025 

3 STRT FIXED BIN, ' ' 00026 
3 L€N FIXED BiNi - , . 00027 

PROF CHAR (10),^ . 00026 ' ? 

• (PA,PB) POINTER,- ' 00029 

FIELDS {^-) CHAP (4) INIT {(4)f4)« »), . ' 00030^ [ 

NUM FIXED BIN, ' x - OOOtJl *i 

RTP CHAR (100) VARYING, ' . 00035 

RQP ,CHAR (100) VARYING, 00033 

HIT.FILE FILE RECORD StQUENT I L INPUT, 00034- 

' CITFILE FILE REQORD btQUENTI L INPUT, " 0'0035 

r ABSNOl OEF ABSNO. * . . • 00036 

2 ABl CHAR (4) \ ' 00037 ' 

2 PAD CHAR (I) , , ' • 00038 

2 AB2 CHAR (6) , , ' '. -00039 ■ 

1 REFNOl BASEL* (P) , ■ ' • 00040 

. .2 GARB CHAR'' (10), ^ , * 6(Hr4), 



I 
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j • ■ 2 «EF1 CriAR (A), 

2 PAD CHAR (1), ' . * • . 

. 2 REF*2 CHAR (6) 

,IPH=^; . 

RDP=RTPI • ' , ' 

ON ENDFILE(CITFILE) GOTO DONE: , ■ 

ON .ENDFILE (HITFlLE ) GO TO ' DO.«EI ' 
'GET LIST (NMAX)» /• CUTOFF*/' 
GET HST(PROF,»NUM) I /♦ Pf^OFlLE MUM8ER* NUMBER OF FIELDS 
. . . tXTRACTFD. IF HiTFILt IS NOT USED.? 

THEN PR^F WILL EQUAL 'ALL* •/ 
PUT PAGE EDIT ('PROF ILt: • .PRO^i 'CUTOFF ; » »NMAX) (SKIP, A» A) I 
/■ PUT gKIP EDIT(»FIELDS: «)(.A); 

GET LIST( (FIELDS(I) DO 1=1 TO UM) )-.» /•FIELOS TO BE EXTRACTED*/ 

PUT skip; 

. PUT LIST{{FIELOS(I> 00 1=1 TO NOM) ) ; 
, A ' IF PR0F=,»ALL:» then ALL=»liSt 

ELSE ALL=«0"»BI 

;■ LOOPl: 

' IF nALL THEN DOI, /♦HI FFILE US-D. READ UNTIL CORRECT PROFILE 

FOUND ' ' / V «/ 

• READ FILECrilTFlLc^) INTO( IT_REC) ; * 
IF PROFNUM-.=PROF THEN GU TO LO'"Pi; • . . ' ' 

END! . . • . 

N=N*H 

. IF N>NMAX THEN GuTO DONE: 
LOOP?; READ FILE (CITFILE ) S'i7>{P)| 

IF -^ALL THEN , Dot /•REAU UNTIL ^ITATION AND HIT FILES COINCIUEV 
IF Aai>REFl THEN GO TO L00P2I 
„ IF AB1=REF1 THEN DO; 

IF AB2>REF2 THEN GO TO' L00P2: . ^ 
- IF AB2<REF2 THEN GO TO LOOPl: 
^ E>4D;. 

IF ABKREFl THEN GO TO LOOPl J 

END ; ■ 

PA=P; /* POINTED TO CITATION RrCOKD*/ 

PB=ADDR(Hlt_LlST) ; /^POINTER Tt POSSIBLY BLANK HIT LIST«/ 
CALL TERMER(PA,FlELDS»Pfa,RDP); 
GO TO LOOPl I 
done: . 

IF IPH=1 THEN do; /»bE-jlN SECOuND PASS*/ 

• PUT page; ' 

IPh=2; 

N=o; 

CLPSE*FILE(HITFILE) ; • • . 

CLOSE FILE(CITFILL) ;. 
PB=NULL; /* si jNAL end of ^IRST PASS »/ 
CALL TERMER(PA,FlELDS»PbfRDP); 

GOVo LOOPl ; • ' ^ •> 

enu; ' • t'' 

pA=.NULL; /» SIGNAL END OF PROCf-TDUHE »/ 
, CALL TERMER(PAtFlELDS»t'B.RDP) ; 

-V END EXTRACT! 

- ^PROCESS . ( • ATR » XREF • ) ; 

termer:. PROC (PP»FlLDS«PTiRTp) J 

; O ' ^ . ■ ' - 

• ERIC 



00042 
'060<>3' 

00044 
: 0004S 

00046 

00047.., 
i 00Q4B' 

00049- 

ooosb • 

00 051 
00052 
00053 
00,054 
00055 
00056 

ooosV 

00058. 
•OOOby 
0006l>- 
00061;. 
000.^2 
00 061 
0006<i 
00065 
00066 
000b7 
OOOoS- 
' 000O9 
/00070- 
00071. 

000^2 

0*0073 

00074 

00075 

00076 

00077 V 

0007B 

0007V • 

OOObO 

oocui 

i OOOB^ 

00083 : 
! 000ti4 
I 00065 
\ .00086 
! 00087 
! 00068 
000B9 
• 00090 
; 00091. 
! 00092 
00093 

ooa94 

00095 
00096 
00097 
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I 

VAR,- 
I>^lf (»0«d) 
(•0«d) 
{•0«B> 
('»l«b) 



INIT 
INIT 
INIT 



$TATC» 
STAT.C* 
STATtC* 
STATtC* 



— / 



DECLARE 
RTP CHAR (100) 
MOSSW BIT (1) 
. NOSSW BIT (1) 
SECSW BIT (1) 
SECON BIT (1) 
(PP,PT) POINTERf • . 
FILDS (^) CHAR iu) 1 
■I CIT^REC BASED CPU) 

d. UNO Char (lO) » 

" 2' REFNO CHAR "( 1 1 ) t 
2 PAD CHAR" (1) , 
2 DAT FIXED BiNf 
2 LCD FIXED BiN'f 
2 LOP FIXED BiNf " • . ^ 

2 DIR (I) , " . ' ^ - 

3 TYPE XHAR {^) » • ^ • - • / 

' 3 STRT FIXED BI^i . ' 

3 LEN FIXED eiNf , j» 
PHI BIT(l)- ALIGNED STATIC INlT{.»l«B)f ^ 
RECNUM>xXED BINOI ) STATIC" In-IT ( 0) , ' 
StRNG CHAR (^000)^ BASEO ^PRJ , /♦STRING FQ« CITATION- RECORDS*)' ' 
STR CHAR (79) BASED (Pt) » /•$T..ING FOR HIT KECORDS «/ . 
ALPH dHAR (26) INIT ( ' ABCDEFG.,IJKLMNOPQRSTUVWXYZ • ) » 
WORD CHAR (^0) VARYlN'-if * ' ' . 

WRKSTR CHAR (2000) f - ' , • 

WRKST (1500) CHAR (1) iJEF WRKSrR* *" ■ • - 
(OtLAST) POINTERf ' • •> ' 

PUNCH FILE STREAM- OUTPUT f . / ' ' . 

(INDX{2:52) PTRf /•POINTERS • TO TERM LISTS*/ 
FIRST riXED BIN INIT (0)f 

/•NUMBE- OF NON-SINGULAR TERMS*/ 
COUNTEo FOR STORWORU* CHEt;KING . ' . */ 
(0)» /*vUMBER OF UNIQUE- TERMS FOUNi) ♦/ 

THAT Some SUCH FROM INTO BEEN BOOK«)f 



•MW FIXED BIN INIT 
NOX BIN FIXED (15) t 
NUMWRDS FIXED 
BADWRD4 CHAR 
(•WERE WITH REFS MADE 
BADWRD5 CHAR 
BADWRD7 CHAR 
BADWR09 CHAR 



(0) t 
/♦ 

BIN INIT 
(64) INIT 
THAN THib 



'STUDY . AFTER 
t^ETWEEN') f 
SEO DISCUSSES 



(33) INIT CWHIC 
(16) INITCPERCEN 
(31) INIT COISCU 
-B4DWRD3 CHAR (39) INIT. 
(•AND THE FOR HAS 'ARE hAS NOT" ONE USE* MAY')) 
STATICf - • . 
1 REC-STATICf- 
■ 2 NUM FIXED OiN INIT 10) 



THESE THEIR') 
•CONDITION^) f 



2 ONE FIXED BIN INIT (1), / 
2 l<NL FIXED BIN* /*NUMRER 0- 
2 LIST (,igO). FIXED BIN* 
ILTERM BASED (Pl> t /"STRUCTU 
2 TERM CHAR (20) ♦ 

■ 2 NO FIXED BINt /•IN^iCATES • 
2 CNT FIXED BIN(15) » 
2 RECN FI>:eD bin (31) f 
2 NEXT POINTER; . . • 

•RECNUM=RECNUM*1I • 
IF' PTsNU^L THEN DO I 
. PH1=^0«B? 



/"SEQUENTIAL RECORD NUMBEft ♦/ 



ALWAYS SET TO ONE */' 
- TERMS IN RECORD «/ 

E FOR EACH TERM FpUNDV 

•F TERM IS NON-SINGULAR ♦/ 



0009&: 
0009.9 
00 10 6; 

ooloi 

00102^ 
00103^ 

ooi.H 

•001 OS 
00106 

00107 

. oorbs, 

. 00109 
OOliO 

4)0111. 

'0()il2. 
0OH3 

0.0114 

001 IS 
00116 
00117 

00 ire 

00119 
00120 

oo,ii.i 

'00i^2 

j)oi2a 

001241 

00125 

0012^, 

0012"F. 

^0,128, 

0012^ 

ooiidv 

.00131 
00132 
. 00133i 
0(J13.4 
00135. 
00136 
0.0'i 37^- 
00.138:, 

ooiiy 
oo/4(j:' 

00141 ' 

00142' 

Oi)143^ 

OJ014C 

00145 

06i46 

00147 

0014,8 

00149^ 

00150 

00151. 

00152;' 

00153 



CIO 



700E7 



Oi/26/77 PAut 



LOORI: 



. L00P2:. 



LOOP 3: 



HERE:^ 



ERIC 



REC.hiUH=Ot ■ 

RETUJ?NI ' . ' ■ ■ • * 

end;^ . ; , - . ■ . . 

..IF PPsNULt fHEN Go TO PHINTR t ' /♦LAST TIME CALLED*/ 
PQ=PP| /♦POINTER TO CITATION R--COKD */ 

*>R==A00R<TYPE(L0D>1)) ; /♦ POINT:R TD CITATION STRING */ 
IF FIRST «=0 THEN DOl /<»INITIALtZE INDX ARRAY TO NULL ♦/ 
. IF lNDEX{RTPt«NOSi)>0 THEN .NO$cW=» 1 'B; 
IF INDEX (RTPt»NOS.») >0 THEN NOS-.W? • 1 »B; 
IF- INDEX{RTP»«NEWSEC»)>0 THEN .=;ECSW=«1»B} 

^ INDX=NULL? V • • ^ - \ 

^ FIRST=i; ^ ■ • . 

^EN[>r 

REe.NUM=REC.NUM*l} •/♦TOTAL NUMfHER OF RECORDS*/ 

PUT SKIP{2) LISl(ftEFNO.REC;N(J'^) ; ' * 

-<NT=0; /-• NUMBER OF- WORDS IN RECORD*/- 
DO X=I "TO 

IF PIL'DS(I) = »» THEN VjO TO l1: " 
IF-'F.ILDSilV^E* THEN i30; /♦REaD TERMS FROM HIT LIST*/ 
WRKSTR=;PT^>STr; ^ ' • 
PUT SKIP EDIT {» ■ FE' ':«){A-): ' ' ' 

JJ=79; 

JK=n„. * 

GO TO L00P2f ' 

END! . . • • ^ • 

JK=:L00-M ■ 

DO-jsl TO jk; \ 
. , IF FILDSM)=«.Ft» Yf^tN GO T.i 'LOOPai . 

IF TYPE(J)-^=FlLDS{I) THEN ..0 TO LPNU2I 
IF T.YPE(J1=M • ..THEN LEN{J)'=-} /♦LOOK 'aT 5 CHAR OFXUtfEN «/ 
PUT SKIP E0I'T(» •.FlLD3(I)f«:«) (A,A»A)} 
JJ=L€N{J)? 
IF JJ> 999 THEN Jj = 999 »^| 

WRKSTR=SUBSTR(STRNG,STRT(J)., iJ) I /♦TRUNCAjTE AFTEr< 999 CHAKS*/ 
DO K=l TO JJ WHILE{JJ>K)} /♦.XAMINE WRKSTR CHAR BY CHAR ♦/ 
IF SUBSTR(WRKSTR,t(,3)=« t t EN DO;* /♦ SMP'OVER BLANKS^/ 
- K=K * JJI •■^ 
•GO TO UPN03I ' 
ENQv; 

IF WRKST{K)=!t$i THEN GO TO HERr I • * 

IF WRKST{K)<»A^« THEN 60 TO Gpn 3IV»SKIP OVcR NONALPHABETICS^/ 

IF WR{?Sh>^)>«Z' THEN GO TO LPN03; ' 
DO KK=K ♦ 1 TO -JJ-ytHlLc. (HRKST fKK) • f)} 

„ end; /♦look For eW.of ter-.,*/ 
ihk=kk; • . . . • 

DO KK=IHK to K by -1 WHiLEiWRK T(KK)<»A»)I " ENOJ 
KK=KKrK*H /♦KK *IS,LENbTH OFJ RM^/ 

IF NOSSW THEN IF WRKST {K*KK-1) r • S • T^JEN DOl /♦REMOVE FINAL S */ 
WRKST(K*KK-n-« •}'. , . . . - 

KK^KK-II 

END? ' 



IF KK<3 THEN OO'I /♦oKIP OVE. 

K--K ♦ KKJ 
> GQ TO LPN03t « 

END} 



TERMS) OF lENTH LESS THAN 3 ♦/ 



'• 0015S 
i 001S6 

ooiba 
001^6 

00161 

00162: 

0016-3 

OOlfeV;* 

0016S 

0U166- 

00167 

0.0166 

0^169 

00170 

00171^ 

00.172. 

OOi.7'3 

0017A 

00 ns 

00i76 

ooi7r 

00,178 
001s79 

00- ltiO 
00.181 ' 

) 00 182-^' 
04183 
0018^ 
0Q185 

"00186 
' 00187. i 

001- 68 
00189 

pi9d 
00191; 
•00192' 
00193 ' 
T)0194 
0Q195- 
00196 
00197 
00198" 
00199 

00200 " 

00201 i 
00202 
00203 

00204 . 

00205 ^ 

00206 

00207 . 

00208 

00209 



CI 



is 6 
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WORO=SUBSTR(WRKSTt^,K,KK) 

/• CH 



CK FOR STOPWUROS 



•mK=oi 

IF KKs3 THEN NDX=INDEX (8A0WRD3 WORD) ; • 
IF NKs4 THEN NDX=INDEX (BADWR.D4 WORD) I- 
IF KK=5 THEN NOX=1NDEX'(BADWRD5.WORD) ; 
IF KK=7 THEN.NDX=lND£X(bADWRD7,W0R0) I 
IF KK=9. THEN N0X=IN0EX(BA0WRD9 . WORD) ; 

IF NDX>0 THEN DO; /♦WOr^U ON SToP LIST* SKIP OVER IT */ 
KsK ♦ KKI 
60. TO LPNOa? 
END? ' 
IF INDEX (teORD»»S»)>0 T-^EN DO} 
IF NOSSW THEN DOI 
K=K*KKJ 
GD TO- LPN03I 

ENOI . . ^ 

ELSE IF SECSW T>IEn DOt 
IF SECON THEN DQt 

SECON=»0«BI 
K=K-7l 

KK=6; 

WORD=SUBSTR (WORD* 1 ♦&) I. 
END? 

ELSE SEC0N=»1»8| 
. ENOI 

PUT EDfT(WORO) (X<i) »A{KK) ) I 
KNT=KNT*liy» NUMBER OF WORDS rN RECORD*/ 

II = IN0EXULPH,SUHSTR(W0R-»1»1)) ♦ 27 - INOEX(ALPH, 
SU8STR(W0RD,2a) ) I /♦HAS-^ING FUNCTION */ 
IF INDX(II)=NULL THEN DOl/* FrRST TERM WITH THIS HASH CODE 
ALLOCATE LTERMI /♦ ALLOCATE R-. CORD FOR THIS tERM ♦/ 
CNT=1| /« NUMBER OF OCCURANCE.- FOr' THIS TERM ♦/ 
.INDX(IIJ=P1} . 
TERM=WORDr^ 
NUMWfiDS=NUMWRO^— r-TT 
R'ECN=:RECNUM> 
-NO=OI /♦ INDICATES TERM IS SyNGULAR */ " 

NEXT-NULL I \ 
K=K * KKI . ' * ■ 

LIST(KNT)=N0I 
PUT EDiT(« («tNO»t> •) (A,F(3) »A) ? 
■GO TO LPND3I 
END I 

QsINDXdDl /»HASh CODE PREVIOjSLY FOUND */ 



IF 0->TERM>WORD THEN DOI /»TER 
^ LIS- 
ALLOCATE LTERM /«ALLOC**TE RECO: 
CNT=1^ 
TERM-WOROl 
INOX(II)sPl^ /♦ PLACE TERM IN 
NUMWRDS=NUMWRDS ♦ 1} 
N0=0| 

recn=:Recnum; 
list(knt)=no{ 



NOT PREVIOUSLY FOUND* SINCE 
IS IN ASCENDING ORDER •/ 
D FOR THI'5 TERM »/ } 



RONT PF LIST 



00210 
00214 
00212 
00213 
002U 
00215 
00216 
00217 
00218 
00219 
00220 
00221 
00222 
00223 
00224 
00225 
00226. 
0 0227 -:" 
00228 
00229 
. 00230 
OO^Il 
00232 
00233 
00234 
00235' 
0023& 
, 00237 
00238 
00239 

00240 

00241 

00242 

00243 

00244 

00245, 

00246 

00247- 

00248 

00249 

00250 

00251 

00252 

00253 

00254 

00255 . 

00256 

00257 

00258 

00259 

00260 

00261 

00262 

00263 

00264 

00265 



ERIC 



ci 
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UPH03: 
LPNU2: 
■LPNDJi 



next=q; /♦link to next kecord tn list «/ 
K=K- ♦ kk;. . 

PUT EOIT(« (•«N0»«) ») (A»F('3)»A) ; . ^ • . - ' 

•GO TO LPND3? 
ENOI 
Q=INDX(II)| 

IF Q->TERM=W0RU THEN DOt 

IF ?H1 THEN IF REGN'iM-.=Q->RECN THEN DO; 

• Q->CNT=Q->CNT*- ; • ' . \ 

IF Q->NO=0 THEN DO; /*WORD P EVIOUSLY FOUND* SHOULD 8£ MARKED 

AS N0N.-IN6ULAR »/ ' I 

NW=NW*1{ - 

Q->NO=NW; ■ ' t 

end; ' ■ ] 

END J • i 

LIST(KNT)=Q->NU; 

K=K ♦ kk; 

PUT EDlt(»(>»Q->NO»')«) (A»F(3),A); 

go to lpnd3; 
end; 

IF Q->t£:RM<WORD THEN DO; /♦WOR MIGHT EXIST FURTHER ALONG THE 

. LIST-*/ 

IF Q->NEXT=NULL THEN ja;/»AT rND OF LIST* SO WORD DID NOT 

OCC iR PREVIOUSLY V 
ALLOCATE LTEBM; /«ALL0CATE RECORD FOR THIS TERM «/ • 
CNT=i; 

Q->NEXT=Pl; /•PUT TERM AT END oF LIST */ 

term=word; 

. . NUMWRDSsNUMWf^OS ♦ 1; 

N0=o; . 

RECN=RECNUMJ . 

next=null; 

4.IST(kNT)=N05 

K=K ♦ kk; • 

PUT EDIT(» (•»N0»e) M {A»F(3)»Ai ; • : 

60 TO LPND3; , 

end; 

last=q? /♦ continue looking '^0«n list for term »/ 

Q=Q->NEXT; 
60 TO L9; 

end; 

ALLOCATE LTERM; /♦TERM NOT F0UmD» PLACE IN PROPER LOCATION »/ - 

' CN.T=i; 
. term=word; 

LAST*->NEXT=P'1 ; /^LINK TO NEXT rERM »/ 
NEXT=QJ /♦LINK TO PREVIOUS TER.- »/ 
NUMttRDS=NUMwROS + U 

NO=0; . 

RECN=RECNUM: . 
LIST(KNT)=N0; 

PUT EDIT(» (»",N0»») • ) (A,F(3) ,A) ; j 

- K=K ♦ kk; 

END L00P3; 
END L00P2; 
END LOOPl; 



00266 
00267^ 
0.0268 

' 00269^ 
00270.-- 
0027L 
00272: 
00273 

rO027-4 

0027^ 

0C276- 

00277. 

00278 

00279- 

00280 

00281- 

00282. 

00283 

00284::. 

00285V 

00286 

0028t 

00288 

00289. 

0029Q. 

00291 , 

00292 

00293 • 

00294 

00295 

-00296 

00297 

00298. 

00299. 

00300 

00301 

00302= 

00303 

00304 

00305. 

00306' 

00307 

00308 

00309 - 

00310 ' 
0p311 
00312 
00313 
00314 
00315 
00316 
00317 
00318 
00319 
00320 .5 
00321 



013 



138 
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ir PHI THEN RETURN} /♦ONLY PRT T ON SECOND PASS. «/ 
PUT FILE (PUNCH) EDIT {REC.NUMi.^NE.KNT) 
(F(3) ;F(3) .F(3> ) 5 
00 K=l TO KNTI /♦ONliY EXECUTED ON SECOND PAS SO NON-SINOULAR 

TERMS CAN BE 'USTIGUISHED. S INGULAR ' TE^Ms WILL 
SHOW UP AS ZE OS IN THE LIST »/ 
PUT FILEtPUNCH) EDIT<R£C.LI T(K))(F(3))} 

end; ' ■ . . 

PUT FILE {PUNCH) SKIP! 

■return; • " . ^ ; 

PRINTR: /♦ ONLY EXECUTED LAST TIM TERMER IS CALLED */ / 
PUT PAGE; 

^ PUT FILE(PUNCH) .eontO.O.O) (F 3) .F(3) ♦F(3) > ; ^ 
00 1=2 TO 52; 

IF INDX(I)=NULL THEN 60 TO L ; : 
Q=INDX(I); . i 

DO WHILE (Q-.=NULL) ; / 

PUT SKIP EDIT('J->N0.Q-.TERM,Q->CNT') (F(3) ,X(2)/,A, 
X(2),F(3)); / 
PUT FILE (PUNCH) SKIP EDIT (Q->Im0,Q->TE.M) (F (3) .A (20) ) ; / 

/» 0->N0 WILL BE 0 FOR SINGULA; TERMS AND A POSITIVE INTEGER 

for non-singular tekms ♦/ 
q=q->next; 
end; 

• . LP: 

end; 

put SK1P(3) edit ('TOTAL TERMS: • fNUMWRDS) ( A,F (5) ) ; 
put; SKIP (2) EDIT CNON-blNGULA*^ terms: • »NW) (A»F (5) ) ; 

return; 
END termer; 



00322 

00323 

0032i!^ 

00325 

00J26 

00327. 

00328 

00329 

00330 

00331 ' 

00332 

00333 

0033^ 

00335 

00336 

00337 

00338 

00339 > 

003^6 

00341- 

003^2 

00343 

0Q34^ 

003^5 

00346 

0034> 

00348- 

00349 

003bO 

00351 
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SInORh; PROCEDUKE UPTIONS (MAIN)? 
/* rOS CLUSTERING. C63^^5» wt HAVE ON TaP^ 
THE CA VOLUME, -b SECTI.^N 1 
HtCORDS LOOK LlKti 

TERMA«"bt.;:TlOf;inF 
IN THIS PROGKA" Wt NOR 



i 



ACCORDING T3 T L MAX T 
THE RECORDS- WRITTEN TO 
TERMa— bt'-l lONl (N 



EU) ,S 
ALIZE 
RMb 
TAPE 
RM FR 



section! (NORM F .EQ) 



OECLAHE 1 OLDWRu -aSED {PiR)y 

2 NPOST KI'XED BiNnS), 
2 WORO JHAR (30 1 
2 
2 
2 




{ I S 1 6,90 ♦ SRp/CASt C . -iOZ II 
INVERTED ON CA SEC/lON/NUM^E^ J 



CTIONtf (FREQ) ... 
THE F.RcQUENCY OF 
Ihl ANY yiVEN SECT! 
_00K like: 
IQ) tSECi I0N2(Npl^H 



CTIONI (FRr.Q) 
H TERM bY bEC]IOi 
IN (dUdS In FHI^ C. 



FREO) ,,,, , 



I 



«SRP»I/ 



FREU KiXED dl 
MAX(loU) OEC 

post(l refer 
trmfrq(oO) dec /i 
j»k bin fixed (15 
casec picture '99 

UNQWRD FlLt RECOR 

OUT FILE -^tCORD S-UUENT/aL OUTPUT; 
ON EnDF I Lr {.UNQWRD > 60 /fo UONE ; 

READ IN TOTAL FRE(JUENCY 
/* FOK EACH SECTION. 



lAED {b*d)*y 
OLDwRD, NPOST) ) CHAR (6) 
ED (ttd)*/ 
INIT (0)> 

/ ' 

seqDeaItial- input, 



TERi 



DO Ul TO oO; 

GET LIS1 (TRMPRO.-D) ; - 
TRmFRO { i ) =8088/ '^RMFRO ( I ) J 

end; 



OFj 

»^ANT TO 

/* NORMALIZE BY MAX TERMS TiHAT 
/* APPEAR IN ANY SECTION 



/* PRINT OUT TABLE UF NORMALIZED 

/♦ FACTORS */ 
PUT SKIP -UITCCA.EC».,.NOHM FACTOR CASECf* NORM FACTOR 

(A( 10) f'^COL (50) lAdO) tA) ; , i 
DO 1 = 1 TO' '^Ol I •■ 

PUT SKIP ^-DIT(I,T.-MFRQ(I) ,I*40»TRMFRQ(I*40)) 

jF (6) »C .L(ll) ,F,6>5e) ,COL(50) »F(6) ,C0L(66> »F(6,2) ) ; ' ' 

tNo; 



HEAD: 



/* READ A BLOCK CONTAINING A TERM 
/* ANU UP TO 100 P05TINGS 

J = 0 » 

READ FILE (UNQwRDi StT (PTR) J : 
^^^^^l' /* COUNT . BLOCKS 



INCRM: do K=1 TO ULDWKD. PUSTi LOOK AT ALL POSTINGS FOK TERM 

CASEc=SUb>TR(OLDW D.POST (K ) , I . J) ; " ^ 
MAX{K)=MAa(K)*TRM RQ(CASEC); 
END? 

> WRITE FILMOUT) F OM (OLDWWD) « 
GO To RE A , i 

DONE: PUT SKIP '--UIT(J,» BLOCKS OF RECORDS PROCESSED » ) (F ( 6) , A) J 
END SlNUR-'i 



C15 140 



S^SuPtW.'PROCEOURE" OPTIONS (MAIN) ? - ■ 

' eiGbEST/TOTAL=Fl - ' . 

FOUR ARKAYS' OF 100 PQSniONs A'HE OEOLAHED: 

FOR Ibi iIGGEST ^tAK >MIN FREQUENCY OF^ OCfcuRA^iCES 



1ST 

2N0 
3R0 



VECTOR 
VECTOR 
VECTOR 
•VECTOK 



IS 
IS. 
IS 
IS 



FOR 
FOR 
FOR 



1ST 

2NU 



THr APPROPIEATE SPOT IN A^^R 



DECLARE 1 WRD BASED (PTR) 



-*I6GEST ^£AK >MIN FRtUUENCY OF OCCUkanCES 
•IGGEST ^EAK <MlKi FKtUUENCY OF OCCURaNCES 
'iOGEST -EAK <MIN FREQUENCY OF OCCUKAIMpES 
Y IS iNcREMEivjTEO BY UNE FOR EACH ENTRY. 

•»SRP*/ 



DECLAr^E 



t ^ 



d 
■'d 
' 2 

d 



(1^) 



NPOST FIXED BIN 
WORD CHAR (20) , 
FREQ FIXED BIN (lb) » 
MAX(IOO) DEC FIXcD (6,2; ♦ 
POST(L REFER (NPOST)) CHAR(6)', 
•L'ASTWORD CHAR (20) INI f (• 
(LCASECtCASEC) PICTURE '999», 
(Fli^STSECSECSec) PICTjKE •v99«, 
(MlNFREQtNUMdLK), DEC FIXED (6^. 
SUPtR(5) DEC FIXED (o,^)» 
(FIKSTPAK,SECPAK,WRPCNT) UEC FixEO 
{F1.F2) DEC F.IXED (b) ♦ 
SW bIT (1) INIT (iO«b) . 

^K.'^c^. ■ ^* COjNTER. FOi< SUPtR SECTIONS ^/ 

IN FILE RECORD SEqUENJIaL INPUT; 

(ARRAYKIOO) .ARRAYS (10.') « ARRAY H 100 ), ARRAY4 (' i 00 ) ) 



DEC FIXEO (6) 



DECLARE UNDERLINE CHAK (6b) J 



ON tNDFlLE(lN) GO TO U )i^jE f 

OPEN FIL£(SY^PRINT) .^T-VCAM OUT-^UT ^^RINT PAGtSIZE'b6) 
LlNESIZt(132) ; 



UOOUl 
000p2 

oooob 

00004 
OOOUS 
00006 
00007 
DOOOd 
00009 
CUOlO 
00011*- 
•006l2.. 
' 00013 , 
00014 
00015 ., 
00016 
00017 ■ 
OOOlb . 
00 01'9 
00020 
OOOifl 
00O£:2 
0002J 
00024, 
0002b 
00026 
00027 ^ 
0002B 
00029 
OOO^JO 
OOOJl 
00032 
00 033 
00034 ' 
000J5 
00036 

00037 . 

00038 , 
00039 
00040 
00041 
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Oi/27/77 PA6t 



OMU69f^03 



INCKM; 



ON'tNOPAGE (SY^OKINT) -^cGlN} 

IF -.SW THE^ UO; 

PUT PAOe? . " , ^ 

PUT 'EDIT CIST diGGtST S'^ND ilGGEST* 1ST BIGGEST', 

•2ND BIGGEST* ) (COL (3 - ) , A,COL (49) ♦A,C0L(9S) ,A»C0L(114) ♦A) ; 

°UT skip; 

OUT fcOIT( 'WOr^O' ♦ 'TOTAL' » •SUPEPSEC", 'SUPEPStC ♦ •%' ) 

(A (3) tA/ai) ,A(a) ♦A(ii)-,x(i) ♦Ad) ,x(6) .A(ii) ,x(^) »A(i) ) ; 

=UT- EO I T (^' WOPD » , ' TOT AL • » ' bL-'PERaEC '♦'%.»,' SuPtPSEC »,•*') 
^^j^^"3) »M21) ♦A(b) rA(l i) ♦X(i).,.4(l) ,X-(6) .A( lU ♦X(2) ,A(1) H 

°UT El)lT{UN0tPL4^E»U.NiDcKHNt) (X(3) ♦A(66) ,X(J) ,A(60) 
PUT SKIM{ " . 

; end; 

END 5 



GET LIST(NUMriLK,MiNFP£v) ; 

PUT SKIP EDIT(NUM8LK, • dLOCKS T(j d£ PROCESStO* ) (F (b) , A)-; 
PUT SKIP EOIT(MINFREU»' IS MINIMUM FREOUENCr ' ) (F (6) ♦ A) 



WRDCNT=0{ SUPER-o; 

ARHAY1 = 0'; ARRAY2=0; ^hPAY3 = 0^ 

fipstpak=)q; secpak=oj 

PIRbTSEC=0; SECS£C=0! 
T=0: 

JNOtRLINE=« 

'; 



array4=o; 



SIGNAL ENOOAGE (SYSPRlNf) ? 
READ FILt(lN) SET (PTk) ; 
1=1*15 

IF I>NUMdL< THEN GO 10 UONE; /* PPOCESStO cJviOUGfi?' 

00 K=l TO NPUST; /« REaO all POSTINGS IN BLOCK 

CASEC=bUBSTfr(POsT(»<) .1,,!) ; /* EXTRACT ONLY SEC NUM8E+^ 

IF CASEC<=20 THEN M= 1 J 

E SE IF CASEC< = 34 FHiiN M = 2; 

ELSE IF CAbEC< = 46 7H.-,\ m = 3; 

ELSE IF CAS£Cv, = 64 TH-- ,x M=4? 

Else if casec<=80 fh-.n m=6; 
super (m)=super(m) ♦max (o ; 

WKOCMT=WRDCNT♦MAX(^) f /•» COu«^jT NUMbErt OF WORDS FOR TERM 



«/ 



IF SUPER(M)>FIRSTPAK J tuM 
IF M-.=FIRSTSEC THEN OW- 
SECPAK=FIRStPAK; 
SECSEp=FIRSTbEC{ 

end; 

FIPbTPAK=SUPER (M) ; 
FrKbTS£C=M; 

end; 



DO; 



-ERIC 



00042 

00043 

0004<f. 

00045.; 

U004& 

00047 

00046 

00049- 

OOObO 

00051 

00052 

00053 

00054 

.00055 
00066 
00P57 
00058r 
00059, 
00060 
00061 
0006^, 

'00063 
00064 i 
00065 
0(J666 
00 06 7 
0006«. 
00069 
00070 
000 71 
000 72 

oo.p/** 

00075 
00076 
00077 
00078 
00079. 
OOOttO- 
000B1-, 
000B2" ^ 
-00083 
00084 
OOOeii 
00086 
00087 
OOOob 
00089 
00090 
00091 
00092 
00093 
00094 
00095 
00096 
00097 -1 



•C1142 
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J 



DONE : 



LAbT WO.'<D=WO>^0; 

PEAU FlLt(lN) SET (PTW)? /« RE.vD NEXT BLOCK ♦/ 
I = I.*1' -. ■ /* COUNT >NUM8£H OF BLOCKS 

..IF I>NUMttLK THEN GO TO UONEl /» PROCESStO ENOUGH? 



«/ 



/* IF THIS BLOCK lb OF SAME WO^^D AS «/ 
/* LAiT CONTINUE INCREMENTING " 
IF LASTwOHD=W0RD ,THEN jO TO IHCRM; " ' " 



('1=FIRSTPA</WRDCNT*100; 
F2=btCPAK/wRQCNT*106? 



/« THi: FOLLOWING DOES ACTUAL ' ♦/ 

/* ADDITION INTO AkRAYS «/ ' ' 

/» COMPUTE 1ST PEAK FOH THIS TtKM 

/* COMPUTE 2ND PEAK FOR THIS TEKM «/ 



IF wRDCNT>=MiNFPEQ Tht i 
AkRAYI (Fl)=ARRAYl (KD 
aKRAY2(F2)=ARRAY2(f-2) 

ENo; 
ELSE 00}" 

AKRAYJ (Fl ) =ARRAy3 (1= 1 ) 
ARRAYS (F2 ) =ARRAy4(P 2 i 

end; 

put edit (lastword) (x (j) 

PUT EDIT (WRDCNTtFlRSTS?: 
(6,2) ♦X(<f) tFO) ♦X(-» 
FIRbTSEC=0; SECSEC=o;' 
•FIkbTPAK=0} SECPaK = ci; 
WROCNT=0; SUPER=0J 
30 TO INCRS; 
''•JT PAGE EOITCIST PtA-^ 
•1ST Pf;AK< •♦MiNFRtO. 
SW=« 1 'e? 

00 j=i TO-loo; 

PUT SK.IP tOIT (ARRAYl K 
(A{b) ♦F(6) »X(10) ) ; 

ekd; 

PUT SKIP EOITd , • BLOC" 
END- S2SUPEf<; 



/* THr" AKRAYS TO WHICH RESULT IS «/ 

/* ASSIGNED DEPENDS .>»HETHEK W.ORU IS */ 

/» LESS ^THAN OK bRtATER TriAN */ 

/* MiNjIMOM FREiUUENLY - «/ 
DO; 

*l; 
*i; 



*i; 
*i; 



♦A{21) ) 5 

L»F1,SECSEC»F2» • ») 
)»F{3)»x{7),F(3)»X{6),F(3),A(l)); 



> • »MINrREQ, »2ND PEAK> ♦♦MInFhEQ* 
•dND PE'.K < '♦MlNFREQ)JA»F(b) ♦X(7)) t 

J) ♦ARRA,Y2(J) tAPRAYJCJ) ♦ ARRAYS (J) ) 

b READ') (F(6) »A) « 



OOOVV 
00100. 
00101 
00102 
00103 

boio^f. 

00 1 Ob 
00106 
* 00107 
00106 
00109 

co-ilo 

00,111 

00112 

0-0113 

OOIH' 

00115 

00116 

0011.7 

OOllb 

00119 

00120. 

00121, • 

00122 

00123 

0012^ 

00125- 

00126 

00127 

00128^ 

00129 

00130 

0U131 

00132 

0013i • 

OOU^ 

0013S 

00136 

00137 

0013d 

0013.9 

00 1 40 

OOl^vl 
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SdSr.": ?ROCtOUKc. OPTIONb (MpIJt ^ ' 

/» ff^r- STEPi OF Tti£ CLUSTEhiINO "I tHM MA-'PING- EXPtWlMcNTS* itc WANT TO FOW 
taCH TEH^^. aUU up ALi. QCCUK iNCtS OF THAT Tt^^M IN ALL CA StC'TIONb 
NO-'MALl/^tO. rH£.\ FIND Trit. ■'iGOtST stCTIuN (CUNTAlNlNb MOST 
OCCUHa.NCcb) ANO COMtJUTE: 

. .nI6oEST/10TAL=Fl 
F^uK A^f<AYS OF loo POSITIUN^ i^HE OECLAf^tO: 

1ST VECTOtf IS FOH is'l •^iOGES.T ^tAN >MIN FkEUUENCV OF OCCUKAnCES 
<fNO VECTOK IS FOR ibUEbT --tAK >MlN FHEuUtNCY OF OCCUKANCtS 

3RD VECTOK IS FOK 1ST - IGOEST -EAK <MiN FRE.JUENCY" OF OCCUKANCES 
'♦TH VECTOK lb FOR 2kJ IGGEST -^tAK <MIN FWEUUENCY OF .O.CCUKjANCES 
THr a::>PK0PIEATE S^'OT Im A^R -Y IS iNCHEMcNTEO BY ONE FOR EACH ENTRY. 



OtCLA'^E 1 WHO dASEO (PTR), ' • ' 

^ NPO.bf FIXEO dJN ( 1-) ♦ 
d WORD C-<hH (20) , 
d FREw f lAtD biN (ib) » 
Z MAX (100) DEC F'l AtiO (fc«2), 

.d P0ST'(L" kEFER (NPUSD) eHAR(b)t • 
" LAbTwORO CHAR (20) In! I (» * •)♦ ' 

tLCASECCAStCi PICTURE 'v^V^ • 
(FI"STSEC»StCSEC) PICTmc. •V^9«» 
.(MlNFREQtNUMyLK) QEC FlAEO' (h)f, 
SEC(eO') DEC FIXED (6*2) ♦ 

(FlKSTPAK»SECPAK,wROCNT) DE.C F I ALU -'(6»2) » ^ 
(F1»F2) UEC FIXEO {&>■» 
SW bIT (1) I.NIT ( tQirt) . 
IN-HLE KtCORD SEQUENTIAL iNPUr; 
OECla^E (AR'VAYKIOO) .ARRAY2{iU.j) , ARRAY <(1uO) »ARRAY4(100)) DEC FJXtD (b) 



DECLARE UNDERLINE CHAR t6b) ; 
ON ENDFlLEdN) GO TO DJ'nE; 

OPFN FILE (SYSPRINT). aT-'LAM OUT-UT PRINT PAGtSIZE (b6) 
LlNESI^^t ( 132) ; 



ON EnJPaoL (sYSPRINT) cloIn; 



»bRP*/ 




Ql/dim PAot 



IF -.SW TMEN OO; 

PUT PAut; /- 

PUT EOITCIST dIGGtbl • ♦ 'dNO -"IGuEST « ♦ ' 1 ST BiGGhST • » 

•iJ\«D BIGGEST' ) (COKSn ♦A.,C0L(46) ♦A,C0L{v3) , A, C0L(112) »A)"; , 

PUT skim; 

PUT eOIT(»wOKD«i«TOTAL'?«CASEC«»«-6'»^,«CASEC»»«*») 

(A(3) tACai) ,A (cj) ,X{1 ! t A(ll) ♦xd) »A{f) »X{6}i,A{li) tA(«i) »A( ) ; 

PUT tpIT(»HOHD«i«TOTAL'»»CASEC»»'^,».»»CAStC» »•*•) 

^("a) ♦A{>1) tACd) .XC 1) lAdl) ,/.(!) ♦A(l) ♦X(0) ,A(li) fAC 1) ) ; 

PUT skip; , ' 

PUT EOIT (U.vj[)EHmNE«UN6r.KLlNE) ( < (3) , A (66) »X ( J) ♦ A (66) ) ; 

•PUT' skip; . 

EN'o; * ' _ — . 

ENO-} 



GET- LIST {M jvi-^LK,MlNFKt v') ; 

^UT SKIP tOIT (NUMBLK. • DLOC^S TO ri£ PKuCESStD' ) (F (6) »A) ? 
pyT SKIP tOIT (MINFREl=^» • IS (-^INIMU'M FKtOUtNCr » ) (F (6) ♦ A) ; 



LCaS£C=000; 
WRl)CNT=0> 5EC = 0; 
ARPay1 = 05 A«^RAY2=0J 



'.KrtAYj=0: A.KkAY** = 0; 



COO'+ii 
00043 
0004/> 
OO'O'vb 
000f6 
UOO**'/ 
000*t6. 

OOO^jO 

ooojjI : 

000S3 
D00b4 , 
■ OOObS 
000t56 
00 0:>7 .: 

000158 ; 

OOObV 

00060 

00,061 

00062- '- 

00063..-' 

-00064 

0006& 

00066 



secpak=o; 

SECSEC-0 5 



f^EAU: 



INCK^^: 



ERJC 



FIKSTPAK=0; 
FI'^STSEC^^O; 

l=o; 

•JNOtKLlNE = » 

"\ 

SIGNAL ENDPAGEXSYSPHIND ; 
^EAO .FILE(IN)- SET (PTK) } 
I = I*i; ' 

IF ,I>NliMtiL'< THEN GO lU UUNE; 



/« PKOCESbEU ENOUGH? 



00 K = l TO VJPOST? ' /« wEaD all PuSTI.VjS IN BLOCK 

CAS£C=bU3STR(P05T(K) .i» j) ; /* tXTRACT ONLj/ SEC NUMBER 

/* IS THIS SECTION J S , HE SAME^^AS 
A THd LAiT bSf^IOl^ ADD TOGETh£»< 
11- CASfC=LCASEC TytN SEC ( C AS^C ) ^SEC ( CaSEC) ♦•'tAX (K) I 

, • " /« OT-iEPWiSE ASSIbi'j FPEUUENCY 

ELSE StC(CASEC)=MAX{^) ; • ' 

wKDCNT = »yRDCNT*MAX (K) /« CO. -Ml NUMBEk OF w0kD3 FOR TERM 

IF bEC(CAS£C) >^^I-fSTPi*K i n£N OOJ- 
IF CAStC-.=Fi;^STPAK TnE bO; 

StC?AK=FlWSTPAK; 

ShCSEC=FIRSTSEC; 

£.\'D; 

FIRSTSEC=CASEC; 
• Fi>^STPAK = SEC{CASEC) ; 

. eivd; ■ ■ • ' 



♦/ 

♦/ 
«/ 

»/ 
*/ 
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oL45 



00067 

000 tib 

00069 

00070 

00071 

000/2 

00073 

00074 

00.0 75 

00076 

00077 

00078 

0O079 

OOObO 

00081. 

00082 

00083 

00084 

00085 

00086 

00087 

00088' 

OOObV 

00090' 

00091 

00092 

00093 

00 09.^ 

00095 

00096 # 

000.97 



OV/27/77 'Pace . 3- 



D>«1069P01 



DONE: 



LCAbEC=CASEC; 
END INCRM; . 

tASTWOHt)=WORl); ^ 

^EAu FILE (IN) SET 1?Th) i /» UE'xU nExT dLOCK \ •»/ 
1=1*1' . • /* COUNT NUMBErt OK dLaCKS 

IF I^NUMdLK THEN GO TO U0NE5 /•» PkOCESSED tNOUl^'^'' 



/» IF TKIS BLOCK lb OF sW WORJ AS 



/» LAiT CONTINUE IWCREMENT\IN6 
IF LASTwOHD=wORO THEN jU TO INCRMI 



F1=FIRSTPAK/.WROCNT*100? 
^2=5ECPAK/wRl)CNT*10„0; 



/* JHt FOLLOWING DOES ACTUAL^ 

/* ADUITION INTO ARRAYS \ 
/» COvipuTE 1ST PEAK FOR THIS T' <M 
/* COMPUTE 2ND PEA^ FOR THIS TERM 



*/ 



/» THh ARRAYS TO WnlCH RESULT IS ♦/ 
ASSIGNED DEPENDS WHETHER wOHD IS •/ 



/».LESS THAN Ow GREATER "THAN 
/* MlvIMUM FREUUENCY 
IF wRDCNr>=MIMFREO TmE-n DO; 
AKRAYl-(rl)=APRAYl(f-il*l; ■ 
ARHf^ ^2 (F2) sARRAVa (Kii) +1 ; 

ens; 
ELSE do; 

• AHRAY3(Fl)=APRAY3(Fi. )♦!? 
ARRAYS (Fe) =A'JRAy4 (F2) ; 

.End; 

POT E0IT{LASTW0RD)(X{3/.A(21,,); 

OUT E0IT(WRDCNT,FlRST5tC,Fl,' :CSECvF2, ' ») 

(M6,2) ♦X(^) ,F(3) »X(o) ,F(3)^X(7» ,F(3) .-X^b) ,F (3) ,A(1) ) J 
FIRSTSEC=0; SECSEC=0; 

FiRbTPA-K-o; secpak=o; 

rfRDCNT=0; SEC=0; 
GO TO incrm; 

PUT PAGE EOITCIST PtA'^> » .MJnpkEU, t;>Nn Pt.AK> «,M2NFREQt 

2ND .'E;»K < » ♦MiNf-REO) (A»F (6) ,X{7) ) ; 



•1ST PEAi^< »,MlNFRr.Q 



SW=M 'b; 

DrO'j=l TO loo; " ■ , , ; 

PUT SKIP*EDIT(ARRAY1 ij) ,ARRAy2(J) ,ARRAY3(J) .ARRAYS (J) ) 

(x(b) fKce) »x(iO)) ; 

Euo; ' 

=UT SKIP EOITd, 
END S2SEC; 



BLUC-xS READ' ) (F (6^ .A) 



00108. 
0010.9' 

00 110; 

00111 

00112: 

001 m,, 
ooiU 

OOllS 
00116: 
0.0117. 
00118 
00X19 

opuo 

, 00121 
.00122 
0u123 
Oqi24 
00125 
06l26. 
00127 
00128 

00129 J 

00130 ■ 
00131 
00132 
00133 
00134- 
00135, 
00136 
00137 
00138 
-00139 
OOUO 
001^1 ' 
001'+2 
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;iNVE.Rt: PROC REORDER 



/* 

OPTIONS(MAIN) } 
/♦ 
/♦ 
/♦ 
/♦ 
/• 
/* 

'. • /• 
/♦ 
/» 
/* 



L' ST UPDATE: 76082^ J 



SRP»/ 



P.OGRAMi INVERT MODULE NO: 
T IS PROGRAM IS THE FIRST ^HASt 
0 THE INVERSION PROCESS-* IT 
a 'LLS OUT EVERY." WORD I.V- THE 
TaUE AND KEYWORDS ELEMENTS OF 
I-TRI-FORMAT RECORDS AND PUTS 
EA- H Wp«Dt WITH THE SPECIFIED 



P0-,TING\ ATTACHEUt 
SO'^TING. THIS IS 



TO A FILE FOR 
A MODIFICATION 



*/ 

*^ 
♦/ 

♦•/ 

♦/ 

♦/ 

♦/ 

♦/ 



OF DM0690^3 FOR CLUSTERING EXPER.»/ 



F0R7NEW format; „S^^« P . . J_UL Y 1974 



CHANGED 
DECLARE 

ONSOURCE BUILT IN t , ' ' 

CHK CHAR (3) STATIC IN?T(« • )'J 

ISTOP- CHAR(24) STATIC INIT(»OF AND THE r'N ON FOR BY'), ' * 

, ' (AVERAGE, DNUMREC,DKOUNTiR)OE FlXED(iO,2), 

1 ORTRY BASED (PLSRP) t - ' 

2 UNO CHAR(IO) , ' 
2 ABST^UM CHAR (11), 
. ' 2 PAD,C^(AR(1) - . 

2 DAT RI'XE-O 'BIN (la) , 

2 LOO FIXED BlN<ia), • . .. 

2 ^LOP FIXED BIN(lb) , ' ■ . 

2 DIRd,). . ... 

J Type .CHAR(f) , *. 

3 ST Fixed -jinqs) , ' . 

3 LN FIXED ►^iNdS), 
•-«(TYP,FT1*FT2) CHAR(f) STATIC, ' ' - ' 

(NUMBER, NUHREC,K0UNT'^,HINUM, Rli^i SIN Fixeb(31), * 

/♦ 'NnTE CAREFULLY THE FOLLOWING ' •/ 
/♦ :OvERLAYS, THEY ARE VITAL IN " •/ 
UNDERSTANDING THE. OaYa MOVEMENT ♦/ 
A-D CHARACTER EXAMINATION ♦/ 
/* R-.UTINES . • • »/ 



LlSri ,CKAR(1000) STATIC lNlt(-' 
ARRKIOOO* CHAR(l) DEF LlSTl, 
LI,ST2 CHAR (255) BASED (LPTr) , ' 



) 



00001 
00002 
00003 
00004 

ouoos 

00006 
00007 

ooooa 

000U9 
00010 
00011 
' 00012 
00013 
00014 
: 00015 
00016 
00017 
00018 
00019 
00020- 
00021 • 
.00022 . 
00023 
00024 
Q0025 V 
00026 
00027 
0002H 
00029 
00030 
00031 
00032-- 
00033 
00034 
, 0003b . 
vO 0.036 
00037 
00038 
00039 
00040 
00041 



/ 
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STATIC INIT{0) 



'0) 



)m6700'»3 

ARR2(1) CHAR(l) BASED (SPTR), 
-XNOX01f>|DX02tNOxa3tNDX04) FIXEO BIN(16 
STPT-.FIXEO bIN(16) STATIC INIT(O), 
(LPTRtSPTRtQPTR) PTR*-' • ' - 

- (LCIT,LCIT2) FIXED 8IN(31) STATIC INIT 
:WORDSP CHAR (^C) STATIC IN IT ( ' ' ^ 

WOROX CmAR(20) DEF wORD$P,, ^ ... 

• ■ » WRDi ' * ■ 

2 WORD CHAR,(20 ) » 

2 POSTING CHAR ' 
PHELD CHAR (4), 

FMTFILE FILE RECORD SEQUENT I L INPUT* 

WORDS FILE RECORD SEQUENTIAL^ OUTPUTl ^ ■ • 

DECLARE HOLD CHAR'(20)t /♦ IN.-UT VARIA/^LE FOR.PRINTING LIST •/ 

PTS.W ^ITd) INIT (»1»B) » 
I BIN FIXED' (31) ♦ • 
i TRMFRQ(80) BIN FixED (31), /^ -.RRAY FOR" CA SEC» TERM FREQ i •/ 

' CASEC- PICTURE 999* » " " ^ . . ■ • . 

■PLSSTR CriAR(lOOO) BASEU (SPTR): 

DECLARE DASH CHAR (19) INIT (>— 1-^ Or 

KOUNTR»NUMREC»TRIV=0; 



FREQ- 
TO BE 



•/ 
.«/ 
,•/ 



ON ERROR begin; 

- PUT SKIP(6) EDIT4«ERR0R AT '.f'JKTRY. ABcTNUH) (A»A), t 

• goto endpgmi ■ - ' , , 

end; • ■ 
m CONVERSION QNSOURCE^OI 

TRMFR3=0; - ■ /« INtTIaIiZE ARRAY^ OF TERM 

- - /*R-AO LIMIT ON CITATIONS 

/« PROCESSED 

" ■ - ■ /• A :D FIELDS TO INVERT ♦/ 

afT EDIT{NUMBER»FT1»FT2) (F(6/ .A(4).,A(A-) ) ; 
PUT SKIP EDIT{«LIHIT:»,NUMB£H»» CITA'rlOf^S. FIELDS: •» , 
FTlrFT2) (A'»F(74',A9A,X.(2) »A) ; • -f , 

/• REaD FIEbO TO BE USED FOR POSTING*/ 
. GET SKIP EDIT-(PFIELD)(4('»>); 

PUT SKIP E&IT( 'FIELD USED FOR pOSTInG: ♦ »PFJELD) ( A) J 

, PWT skip; 

. GfT .SKIP' EDIT (HOLD) (A( •/).).} ' 
PUT SKIP EDIT (HOLD) (A) 5 

IF h6lD=»N0OHINT» THEN PTSW=«0-iB; ' . ' ' ' 

ON tNDFlLE (FMTFILE) GO TO ENDP'.m; 
ON RECORD (FMTFILE) BEGIn; END; 
■ ■ . ' /* • 



START 



♦/ 



READ FILE(FMTFILE) SET(PLSRP)i 
SPTR=ADDR(TYPE(L0D*1)) ; 



KOUNTR=KOUNTR*l; 

^£N=Q I 
NDX02=0; 



ERIC 



/« FtRST LOOP LOOKS FOR TITLE 

.C23r48 



00041 
00043 
0'004A 
0004b' 

"00046 . 
00047' 
00048. 
00049/ 
00Q50-. 

. 00051 
0(jp52 
. 00053 
000S4, 
000S5 ^ 
000£6 
00X)S7 

00 ass* 

O0OS.9 

• ooo&o- 

>4lb062' 
J (f0063 
00064. 
. 00065 
00066 
00067'* 
'OOObS 
00069' 
: 00070. 
: 00071 
00072- 
00073 
,00t)74' 

ooots 

00076 

000-77 

00 078- 

0.0079 

00080 

00081 . 

000.82^ 

00083 

00084 

00085 

00086 

00087 

00088 * 

00089 

00090 

00091 ' 

00092 

00093 

00094 . 

00095 

00096 

00097 



MO 70 0^3^ 



01/26/77. -P-A'Ut- 



LOOPOU 

' DO-NOX01= ,1 TO LOD-ir 



/• FlEXDt IF IT FINOS THE K€YWO»<OS '«/" 
/♦ FfRST, IT KEMEmBERS THAT FOW ' «/ 

. . •/ 



/« 'L TER 



FOUND l: 



TYP=TYPE(NDX01) I 
, IF tYP=FTl THEN GOTO FOUOl;.. 
IF TYP=FT2 THEN mOX02=ND <0 1 J " 

En6 loopqil; ... 
go to lo6po2; 

len=ln(ndx01h 



l-LPTR=AOOR(A>^Rl (1) ) t 



/« T E NEXT SECTIO^N MOVES" THE TITLFV 
■/• Tm- a work area to EXAMINE ITt »/ 
/♦ I- A REASONABLY EFFICIENT MANNER*/ 



QPTR=ADDR{-ARR2(ST<NDX'Ji) ) H ^ • " 

IF .LENi<=85 THEN SUBSTR (LPTR->LIST2, 1 ) =SU8S'f R (0PTH->LIST2, 1 i 8b) » 
ELSE IF LEN<=140 THEN . ... ' 

'^■suaSTR(LPTR->LIST2,r»140)=SUBSTR^(QF>TR->LlST2,l'f 1?40) ; ' 
ELSE do; * ■ • s , . 

LPTP->L IST2=QPTR->1I ST2 I 

-IF LEN>25S THEN 001 - - .. 

|lPTR=?ADDR{ARR1 (256) ) ; ' 

<3PTR=ADDR{ARR2{ST(NDX0l)*2bb) ) ; 



\ 



LPTR->LlSr2=QPTR->LlST2; 
I 

LCJT=LCIT*ll 
\ IF LEN>510 THEN. DO; 
LCIT2=LCIT2 ♦ 1; 
LEN=510; 
- • ENaf 
ENO': - " 

•end; 



/♦ I • LONGER THAN 510 tHARACTEKSi 
/♦ T-E FIELD. IS TRUNCATED ,' ' 



«/ 



/» 



»/ 

• - /♦ Ir THE KEYWORD FIELD WASN'T. ^^/- 

/* F.-iUND. BEFORE* IT IS NOW SOUCjHT <»/ 

LCOP02: • 
"^IF NOXn2>'0 ' THEN 'GOTO F0UND2; 

DO NDX02=NDX0i TO LOD-U • - . • 

typ=type(ndx02) ; ' •• 

if typ=pt2 then goto fou d2t 

end; 

;* • • GO TO ONOi ;" ' \> 

/* T E KEYWORDS AKE -NOW MOVED »/ 

„ ' , . /» SIMILARLY TO WORK AREA FOLLOWING*/ 

' ' /» T.^E TITLE . ■ \/ 

fFO^ivf.': ■ ' ■ ■ • ^ 

A^<>i (lEN*1) = » ^« I 

LEN2=LNiNOX02H 
LPTR=ADDR (ARRl (LEN*2) > » 
, - , QPTR=ADDR^ARR2lST(NOX0<f) }) ; 

IF LeN2<sr65 THEN SUBS^TR (LPTR->L I ST2 » 1 , ) =SUBSTR (QPTR->LIST2» 1 ,65 j ? 
ELSE IF LEN2<=110 THEN " . - y 




/ 



C2 



oooi^y 

OOlv'O 
Oolul . 

00102. 

.^OOlOi 
OOiOi*' 
001 j5., 

- 00105-' 
00i;-.>7 
001 )H ' 
001' J » 

! 001 lu -. 

i o!oiii 
; 0O113 

001 
: OOl'lb 
. 00116 

001 1?-. 

00 lid' 

,00119 
" 00120 . 

odi^Ji 

- 00122^ 
00123 

"00124 

• X)0125 ' 
001^6 - 
00127 ^ 
Colts'" 
O0U9 
OUi JO 

^001.11 - 

001 J2 
0-0 l.U 
001J4 

boiJs. 

00136 
00137 
001 3fa 

• 00139 ■ 
00140 
•00141 
0014j 
0014! 
0014^ 
,0014b 
00146 
00147 
0014fc 
00149 
OOl'bO 
00151 
001.52' 
^^53 



t 



•4 



)M070043 
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SU«|^TR(LPTH->LI*ST2.\»nO)::SUBSTR(QPT«->LIST2.1»110) I 
ELSE 00 t - , 

LPTR->LJfT2=QPTR->l.IST25 ' 
IF LFN2>255 THEN 001 ^ * 

liPTR=ADDR(ARRl (LEN*25V)'> 

.^QPTR=AODR(ARR2{ST(NDX02J ♦2d5) ) | 
.LPTft->LIST2pQPTR->LIS: 
. LCIT=LCIT*H 

IF LEN>S10 THEN DO J 
LCIT2^LCIT2 ♦ li 
LEN=510; 
, END? 
END X 

end; 

;-WEN=LEN*UEN2*l} 



/ 



!->LISjP2jK^ 



'I 



. ONO-1 i 

•.-IF LiEN=0 tiHEN -GOTO START I 
. STPT^ir 

; LPTR-ADbR(ARRltl) ) i 
LEN=LENtll - 
.ARRl{LEN)=» •} ; 



/» Now THE- "words MUST 8E BROKEN OUT*/ 
/«■ Of. BOTH TITLE AljlD KEVS ♦/' 



i — 

/• LnQV AHEAD TO A NON-ALPHABETIC '♦/ 
/» C .ARACTER . «/• 



.LOOP03: oi NDX03=l- T.0-1.EN BY U 
IF ARRi (N0X03)>=»/^ THEN GOTO f-L00^3; ^'7 

• . ' /• tke else block checks for an 



•/ 



Else do; 

X£N2=NDX03-STPT; 
■IF LEn2<^. THEN D0> . 
' IF LEN2<2 THEN GOTO NOI • 
CHK=SUBSTR(LIST2»STPT,3> ; 
IF .SUBSTa<CriK,l,.l-)>»Z« THEN 60T0 NOl , 
W0RP=CHK ; / 
IF INDEX (?T6p,CHK)>0 THEN GOTO NOr ^ 
END? * ^ ^ 

ELSE oor ■ ' 

^F ARRKSTPT) >«Z» THEN GOTO NO? 
IF LEN2>=20 THEN WORD=SUBSTR (LIi.T2,STPTt20) I 
■ELSE DO; 

.WOROX=SUBSTR(LIST2,STPTf20r* / 
SUBS TR(W0RDSP»LEN2V 1,20) =» i; 

/♦ EXrRACT POSTING 

• DO NDXOl = l TOu-LOD-i; 
IF TYPE(NDX01)=PFIELO f> PFIELD=«1 « THEN 

POSTING=SUBSTR(PLSbT'<»ST(NDX 1) ♦S) K 
IF TYPE{NDX01)ePFlELD ^ PFIELD-'S '» THEN 

P0STING=SU8STR(F^LSSTHfST{NDX 1)*3»6) ; 

ENOI 

W0RD=W0RDX4. . — 

>'Er!c — ^. ^. ^ c2^5(j 



/* ArCEPTABL,E wOHU, REJECTING IF •/ 
/♦ 0-iE CHARACTER* BEGINS WITH A * •/ 
/« N fMBBR* OR 'f<PPE^RS IN THE QUICK •/ 

/• Stop list #/ 



♦/ 



001S4 
001S5 
00l!>6 
00lb7. 
OOlbS 
001b9. 
00160 
00161 
00162 
00163 
0016^ 
00165 . 
00166 
00167 
.00168 
00169 

ooi?e 

00171 • 
00172 
00173 
00 1 7^f 
00175" 
Q 0.176 
00177 , 
001^8 J, 
-001-79 
•00180- 
OOlbl 
00162 
00183 
0018u 
pai85" 
Oai86 
00187 
00188 . 
Oai89 
0019 
00 

00192 
00193 
00194 
00195 
00196 
00197 
00198 
'00199 
00200 
00201 

002 
00204 
00205 
00206 
00207 
00^08 
0(ye09 



1190 
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Tend; 

I.ENOt 



IF PTSW THEN PUT EDI T ( WORD,POSt ING) ( A (20 ) , A (1 0) ) t 
IF PFIELD=»5 » THEN l'OI 
CASEC=SUBSTR (POSTINGi 1 tB) I 

TRMFRQ{CASEC)=TRMFKQ(CASEC)* I 
END! 



»/ 
♦/ 



♦/ 

«/ 



»/ 



/♦ HE-E WORD AND POSTING ARE 
/* W. ITTEN 
• . WRITE FILE(WORDS') KHOM(WRD); 
NUMRECsNUMREC+l; 

/• S^IP HERE TO GU ON AFTER 
/* REJECTED TERM 

NO: STPT=NDX03*H 
ENDt 

£L0OP3J END LOOP03^_ 

ENDCHK: IF KOUNTR<Nl;^ER THEN -jO TO ST.^RTJ 

/» 

r^,r.r.r. « /» E iD OF PROGRAM* PRINT STATISTICS*/ 

.ENDPGM5 CLOSE FILE (FMTFILE) , FlLE<WORD';) ; ' iM.iDiiua / 

^U^j|DIT( 'NUMBER OF CITATIONS .PROCESSED: • ,KOUNTR) (PAGE, A (30) , 

PUT EDIT (•NUMBER OF POSTINGS • 'NUMREO (SKIP (2) ,A, 
F(8) ) ; 

PUT SKIP (2) EDIT ('FULL LENGTH mOVE USE. 'tLCIT,* TIMES. • ) (A,F (<f) , A) ; 
m SKlP(l) EOITC TRUNCATION OCCURRE NLCIT2. • TIMeL » (A,F (i , A ; 
DNUMR£C=NUMREC| . 
OKOUNtR=KOUNTR» 
AV£RAGE=aNUMREC/OKOUNT'' » 
PUT EOITCMEAN NUMBER OF POSTIN'GS PER -ITATION 
AVERAGE) (SKIP(2) ,A(3/) ,F(10,-) ) ; 

PR 'NT FREQUENCY OlF TERMS FOR EACm»/ 
/« CA SECTION ff , ■ tt/ 

TF PF.IELD='5 • THEN 00 1 

PUT PAGE EDIT( •CAStC^f* , 'TOTA FREQ OF TERMS' f'CA SEC^', • 
^ - 'TOTAL FREQ OF tERf'S • ) ( A ( 1 ^ ) ,A,COL (50 ) , A ( 10) , A) ; • 

PUT SKIP EDlT(DASH*DASH,DASH,DflbH) (AdM ,A(19) »COL (bO) , A ( 10) , A ( 1 9) ) I 
DO 1=1 TO ^01 

PUT SKIP EDIT{I,TR:«lFRQ(I),r*<»0,TRMFRQ(I*40)) ' * 

4F(6) tCOLdl) ,F{o) ♦C0L(5 ) ,F{6) ,CUL(66) ,F(6)) ; 
ENDt 

END» 

END INVERT|>' 



00210 

002U 

00212 

00213 

00214 

0'021S 

00216 

00217 

0021d 

00219 

00220 

00221 

00222 

00223 

0022^ 

O022b 

00226 

00227 

00228 

00229 

00230 

00231 

002-32 

00233 

00234 

0023d 

00236 

00237 

0023b 

00239 

0024y 

00241 

^024€ 

00243 

6o24<» ■ 

00245 

00246 

00247 

00248 

00249 

002b0 

00251 . 

00252 

00253 
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i' 



LAST 
LAST 



update: 
update: 



760630 
751001 



:SQU££2: PROC REORDER 



DECLARE 



1 



0PTI0NS{M>.IN) ; 

/♦ 
/♦ 

/♦ 
/♦ 
/♦ 

/* 



Program: squeeze module no.: 

T OS, PROGRAM READS THE (SORTED)*/ 
P .STINGS FROM THE INVERT PROGRAM*/ 
A^.D CREATES BLOCKS OF UP TO 100.*/ 



T IS IS A MODIFICATION OF 
D 0690«f4 FOR EXPERIMENTS FOR 
Ci USTERING. A POSTING CAN BE 
E'THER A C00EN(6 CHAR ) OR 6 
DrGlTS OF THE CA SECTION # 



KRD BASED (WPTR)» /• WO D AND COD£N NOW SORTED 

2 WORD CHAR (20) , 

2 POST CHAR (6) , 
OLDWRD BASED (OWPTR) » 

2 NPOST FIXED BIN (15), /* NO. OF POSTINGS IN THIS 
2 OLOWORD CHAR (20), 

2" FREQ FIXED BlN(15). /* NO. IN ALL BLOCKS THUS FAR 

2 MAX(IOO) DEC FIXED (6,2), 

2 POST(K R£FER(NPOST) ) CHAR 
K FIXED 8IN(15) STATIC INIT(0' 
(TDUP.DU?,C0L9LlN) FIXED BIN( 
LPOST CHAR (6) ♦ 

(L0,L2,L3) FIXED BIN (31) STArIC INIT(O), 
(STOP(0:122) »H0L0) CHA'^ (20), /« STOP LIST 



»/ 
*/ 
*/ 
«/ 
»/ 



*/ 



♦/ 



6) 



INIT(O) , 



♦/ 



THL CHAR(^^) STATIC 

INIT(«TERH • NPOST 

JTIM CHAR(^»4) STATIC INIT(« •)» • 

PIM(50) CHAR(132), 
(UPP,LOW,DIV»SAVER) BIN FIXEOi 
(TOTAL»NUM,J»L,M) 3lN FIXEDOl)* 
(DJ,DL, AVERAGE) DEC FlXED(10,2), 

(PTSW, ASW) BIT(l) ALIGNED StATIC, 
WORDS FILE RECORD SEvJuENTIAL INPUT, 
UNQWRD FILE RECORD SEUUENTIA. OUTPUT* 
OPEN FILE (WORDS), FILt(ONQWRD) 

OPEN FILE(SYSPRINT) PkINT LIN SIZE (132) 



CITS . FREQ') , 



PAGESI2E(55) » 



ON tNDFILE (WORDS) 
ON tNDFILE (SYSIN) 



K=ioo; 



GO 
60 



TO 
TO 

/<» 



ENDPGM 
CONTIN 
BUILD 
FILL L 



MAXIMUM-SIZE 
TER 



BLOCK TO 



•/ 

.*/ 



/* 



CH 

LI' 



CK WHETHER TO .PRINT FREQUENCY 
T BY READING -CARD FROM SYSIN 



ALLOCATE OLOWRO SET(OWPTR)T 

GET EDIT(HOLO) (A(20) ) » 

IF holds&noprinTi then ptsw=« 'b; 

ELSE/PTfw=«l«B; 
MAX=1I 

LOW, I,ITRIV»NTRI=o; 
TDUP,DUPf LPOST, LIN=0J COL=n 

/*» READ T 



ITIM=« M 
E STOP- Li ST 



*/ 



ERIC 
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00001 . 
00002 
00003 
00004 
00005 
00006 
00007 
00008 
00009 
00010 
0001 1 
00012 
OOOIJ 
00014 
00015 
00016 
00O17 
00018 
00019 
00020 
00021 
00022 
00023 
00024 
00023 
00026 . 
00027 
0002B 
00029 
00030 
00031 
00032 
00033 
00034 
00035 
0M3(i 
00037 
00038 
.00039 
00040 
00041 

00042 
00,043 
00044 
00045 
00046 
00047 
00048' 
00049 
OOObO 
00051 • 
00052 
0(J053 
00054 
'00055 
0005^ 



*/ 
*/ 



- read: get EOIT(HOLO) (SKlP(i)»A(20).)> 

stop(I)=hold; 
60 TO read; 

MCONTIn: savers! ; . /♦ nUvBER of STOP WORDS ^ 

UPP=2; ' 

SET UPoEH BOUND FOR BINARY 
• /* SEARCH (POWER OF 2) 

LOOPOO: 00 WHILE (UPP<SAVER) ; 
UPP=UPP*2; 
END LOOPGOI 

• ' ILIMIT=:UPP| - -=> 

TOTAL»I»JtL»M=0l 
rtEAO FILEtWORDS) SET{WPTR)| 

OLD WRD „ OLDWORD=WRO . WORv i 

/* T.E NEXT BLOCK IS A NORMAL 
/♦ BINARY SEARCH OF THE STOP LIST 
/♦ To DETERMINE IF THE TERM LIES 
/« TME.REIN 

LOW=:0l 
UPP=IHMITI 
0IV=UPP/2; 

IF DIV>SAVER THEN DO; 
UPP=DIV; 
GO TO COMPTI 
ENDt 

HOLD=STOP(DIV) t 
IF OLDWORD<HOLD THEN DO} 
UPP=DIVI 
GO TO compt; 
END* 

IF OLDtfORD>HOLD THEN DUI 
LOW=DIV? 
GO TO COMPTI 

END! ■ . - " 

30 TO ONOO; 
DIV=(L0W*UPP)/2I 
IF DIV ^= LOW THEN 
ELSE 



•/ 



♦/ 



COMPR; 



COMPT: 



ONOO : 
ON: 

UNIQUE 



ITRIV=U 

GO TO READER; 



GO TO COMPR.: 
GO TO ON? 

IF WOR 

ITRIV= 



IS IN STOP LIST. SE,T »/ 



iTRlVsOl 

OLDWRD.POST { 1 ) =WRD.POST } 
KtNUM=l} 

TOUPrTDUP*DUPI DUP=0; 



OTHERWISE SET 
OUTPUT BLOCK 



ITki\/=0 AND SET 



JJP*/ 



LC=Ol 
READER: 

«EA0 FIL£(W0R0S) SETCWPTR); 



/• RE D AN EXTRACTED WORD & POSTING 

/* THS PROCESSING INVOLVES WORDS 
/« WH-CH HAVi. BEEN ENTERED BEFORE 



»/ 



ERIC 



C28 i 4 t 

Id J 



if.wrd.woro=oldwrd.oloword then do! 
asw=»o«b; 

IF ITRIVsl THEN GO TO REAi^tR? /• kKIP IF A STOP WORD 



ELSE 001 



/• CH. CK IF WORD APPEARED IN SAME 
/• PO.-TING ' •/ 



IF WRD.POST=LPOST THEN )0| 
0UP=DUP*l; 

MAX(K>=MAX(K)*i t ENO; 

/• IF BLOCK IS FULL9 WRITE IT 

ELSE DO; 

IF K=100 THEN DOj 

IF L0=0 THEN DO; L2=L2*l; iO=l; END; 

i=i; 
60 TO writer; 

, end; - j 

/• then* or otherwise* set POSTING h 

/ . /** IN BLOCK > <• . »/ 

K=K*i; 

NUM=NUM*l; /• CO.iNTS NO. POSTINGS PER WORD 

IF NUM>M THEN M=NUM; /• HIGHEST NO. POSTINGS PER WORD 



OLDWRD.POST (K) =WRD.'P0ST; 

lpost=wrd.post; 
. j=j*n 
end; 

GOTO reader; 
end; 



end; 

ELSE ASW=«1'B; 



/ 



WRITEW: IF ITRIV-i THEN Oo; 
NTRIsNTRI*!; 
60 TO ONOl; 

end; , • ■ 

IF L0=1 THEN do; . 
L=LAi 
L3=L3+k; 

end; 

npost=k; * 
olowrd.freq'=numi 

WRITE FILE<UNQWRQ) FROM (OLDWRD ^ ; 
IF ASW THEN 

IF PTSW THEN CALL PRNT; 

max=i; 

K=l? 

j=J*i; 



IF I?l THEN oo; 




/• THrS GROUP IS EXECUTED AFTER FULL*/ 
/• BL CK IS WRITTEN. SET FLAG BACK »/ 
/• TO 0 fci POSTING dECOMES FIRST »/ 
/♦ POcTING OF NEW BLOCK •/ 



. 154 



00113 

: 00114 

00115 • 
• 00116 
~ 00117 
00116 
00119 
00120 
00121 
00122 
00123 
0012-!^ 
00125 
: 00126 
00127" 
00128 
00129 
' 00130 
00131 
i 00132 
: 00133 
00134 
00135 
00136 
00137 
00136 
: 00139 
00140 
: 00141- 
: 00142 
: 00143 
00144 
00145 
I 00146- 
: 00147 " 

. 00148 
: 00149 
00150 
00151 
00152 
00153 

^00154 
00155 
00156 
00157 • 
'00158. 
0015.9 
00160 
i 00161 
: 00162 
00163 
00164 
00165 
00166 
00167 
00168 
00169 
00170 



I-OI 

0LDWRD.P0ST(1)=WRD.P0STI 

LPOSTsWRD.POST» 

NUM=NUM*U 

60 TO reader; 

ENDt 



ONOl.: 



COMPAR : 



/ 



COMPUT: 

TR: 

0N02: 



OLDWRO.OLDnJORDsWRD.WOROI 

LOW=OI 

UPP=I'LIMIT; 

DIV=UPP/2I 

IF Diy>SAVER THEN OOi 
UPP=Diyi 
GO TO COMPUT n 
END I 

HOL0=ST0P(DIV) I 
• IF OLDWORD<HOLO'*ThEN iDOl 
UPP=DIVI 
60 TO COMPUT 1? 
END! 

IF OLDWORDjHOLD THEN DO J 

LOWsDIVI 

60 TO COMPUT n 
I END I 
GO TO TRI 
Diy=iL0W*UPP)/2f 
IF DIV LOW THEN 
ELSE 
ITRlV=n 
GO TO RE^AOERI 
ITRIV=0I 

total=total*1i 

oudwro. post .( 1 ) =wro . post ? 

lposx=wrd.p6sti 

go to unique! 



/* •/ 

/♦ 8i OCK ENTERED IF ITRIV=1. AT */ 
/« W-ITER REQUEST OR NEW TERM THIS */ 
/♦ BrNARY SEARCH IS WORD AFTER THE •/ 
/♦ F RST TERM READ */ 



GO 
60 



TO COMPA 
TO\ON02I 



€NDPGm 
IF L0 = 
L=L*I 
>L3=L3 



1 THEN 
t 



00 J 



Np0ST=KJ • 
OLDWRD,FREQ=NUMI 

WRITE FILE(UNQWRD) FROM (OlOWRD) t 
/ IF PTSW THEN CALL-PRNTI 
TDUP=TDUP*DUP; DUP=-H 
IF PTSW THEN CALL PRNTI 
J=J*H • - 
TOTAL=T0TAL*H 

PUT EOIT(»NUMBER OF UNIQUE WOR ^S: • tTOTAL) 
PUT EDITCNIiMBER OF TRIVIAL WOpDS:»,NTRI) 



PUT EOITf'TOTAL POSTINGS: « » J) (qKiP (2) f A ( 15) ,F (10) ) ? 
PUT EDIT ('DUPLICATE POSTINGS .jEMOVED : • » TDUP) 
(SKIP.AyFdO)) ; 



(PA6£»A{23) ,F(8) ) i 
(SKIP(2) fA{24) ,F(8)) 



ERIC 



00171 

0017-2 

00173 

0017<» 

00175 

00176 

00177 

00178 

00179 

00180 

001.81 

00182 

001iJ3 

00184 

00185 

00186 

00187 

00186 

00189 

00190 

00191 

00192 

00193 

00194 

00195 

00196 

00197 

00196 

00199 

00200 

00201 

00202 

00203 

00204 

00205 

00206 

00207 

00206 

00209 

00210 

00211 

00212 

00213 

00214 

00215 

00216 

00217 

00218 

00219 

00220 

00221 

00232 

0022^ 

00224 

00225 

00226 

Q0227 
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0L=T0TALI ' . 

AVEHAGE=DJ/DLJ 

PUT EDITCAVERAGE.' NUMBtR OF POSTINGS PER WOHO: AVERAGE) 
(SKIP{2) tAOb) ♦F(10,>) ) I 
DD=TDUP; 
AV£RAGE=DD/DLI 

: . PUT EDIT (t average: number of O'JPLICATE POSTINGS:', 

AVERAGE) (SKIP, A, F( 10.2)) I 

J=J-L3J 

PUT SKIP(5)' EDITCTOTAL LoW-FRtUUENCY POSTINGS • , J) ( A,F (a) ) ; 
D J= J X " . . ■ . 

DL=T0TAL-(NTRi*L2) t 
AVeKAGE=DJ/DLI 

PUT SKIP (2) EDIT ('MEAN NUMBER OF POSTInOS PER LOW-FREOUENCY, NON-TRIVIA 
L WORD AVERAGE) (ArFtlO, 5)) i 
PUT SKIP(3> » ■ • . 

PUT E0IT(,L2,» UNIQUE MIGH-FREQUENCy' WO^OS • , — 
Lf • TOTAL RECORDS FROM H-F'WoROS«< " 

L3,t TOTAL POSTINGS Fi^OM H-F -ORDS • ) ( (3) {SKIP (2) ,F (8) , A) ) ; 
PUT EDIT < 'HIGHEST NUMBtR OF PG-TINGS P£R waRD:',M) 
. (SKIP(2) ,A(36) ,F(8) ) ; 

/* the .rnt subroutine is used to •/ 
/•print the term frequencies in a •/ 
• k /'three column per page format. */ 

ppnt: proc reorder? 

DCL li,I2 FIXED BIN (It:') I , 
flush: PROC I " 

PUT PAGE EDlT(THL,THL,THf ) (A(44) ,A(4^) ,A(4<t) ) I 
- DO 11=1 TO 501 

PUT SKIP ED1T(PIM(1)){A(132)); 

• END! 
END FLUSH! 

IF DUP<0 THEN DO; /* FORCED rLUSH AT END OF RUN */ 
DO I1=C0L 10 3? 

DO I2=LIN*1 TO 50; . 

SUBSTR(PIM(I2) {44«Il)-43,44)=t i; 

END I 

LIN=0« r— 

end; 

call flush? / 
' return; 
end; . 

LIN=LIN*U 
IF LIN>50 THEN Dq; 
COL=COL*l;. 

IF C0L>3 THEN DO; • " 
; CALL FLUSH I 

coL=i; 

■ end; - ' ■ 

LiN=i; 

enui 

PUT STRING(ITIM). EDIT (OLDWORD .NPOST ,FF*EQ,FHEQ*DUP) 
(A('20) , (3) (F(7)p i • ' 

SUBSTR (PIM (LIN), (44»CUL) -43,4.) =ITIM; 
END PRNT; 

<^ ■ END SQUEEZI _ 

ERIC .15 b 



00228 
00229 
00230 
00231 
00232 
.00233 
00234 
a0235 
00236 
00237 
00238 
00239 
00240 
00241 
00242 
00243 
00244 
9^45 
"00246 
00247 
O02'-f8 
002-+9 
00250 •■' 
00251 
00252 
00253 
00254 
00255 
00256 
00257 
00258 
00259 , 
00260 
OOt^l 
00262 
00263 
00264 
00265 
0 0266 
.00267 
.00268 
00269 
00270 ' 
00271 
00272 
00273 
00274 
00275 
00276 . 
00277 
00278 
00279 
00280 
00281 
00282 
00283 
002^4 
00285 
00286 



C31 



/» LAST UPtlAlE: 750103 •/ 

cluster: 

- PROC (RTP) REORDER OPTIONS (MA' N) I 
OCL ^ ■ . 

RTP CHARdOO) VAK» ^ • ^ 

1 OISTNODEt . 

?. TERMl FIXED BIN(15)? " " ' . 

2 TERM2 FIXED BIN(15)» * 
2. TTDIST FLOAT .DEC(^))» ■ " ^ .. 

WRITESW BIJ(l) ALIGNED* - . ' 

Si^G BIT(l) ALIGNED STATIC INtTCI'B)* ^ 
J)UT FILE REC0aOf._ , - 

LNECl FIXED BIN (15) STATIC* - - V \- 

LN.(200) fixed BIN(15) STATIC, 
'ELTS.(lIMAXf2) FIXED BIN(i5) CTLt-i.. ^ 
NXT(0:TOP) fixed tJiNdS) CTL* - ' 
FORM(200»100) CHAR(l), 

FQSMOJD.CHARdOO) .D£F FnRMt ' v 

(IIMAXtFTRSTtLASTtCUR.ECi tECS) FIXED BIN(i5) STATIC? 
AS50C(200) DEC FL0At(9)9 /• ?ASS0C WITH ABSORBER 

' GRiiUB (20.a) , F I XED. B I w ( 1-5 ) t/.f NO , OF ABSOEBER fl 
SIZE (200) FIXED 8Im(15)*/» SIZE OF GRt)UP 

d6CTRM(1OO*O:4O0) FIXED rIN^(15)*/» DOC-TRM COIN ' */ 
DjDij:DaeXQ15Qoa) D_£C fL0All.6)» /♦_.OOC-D.O.C ASSd^d ARRAY 
CURR(IOO) FIXED Bl'u (13) ♦/♦NO.OF CURR' USER OF ROW 

( I ♦ DOCMAX ♦ DOCNO ♦ TCNT » TRMM aX » TRMNO ♦ MULT 1 ♦ MULT2»i,J • LOC »" . 

QQCCNt, _ „ . 

T*INT»UN*HI,HJ*T0P,I2*BAsX*BA'sf*BASJ*8ASk»L0C2*L0C3* " 
HHJtrtHi) • FIXED BImOI) STATIC INIT(O)* 

(Xl.».JW>D5IMlN»DlST»DIJ,DJ_<,DiK,.D.lX»ALPHJAALPHK< ^ 
BETAtGAMMA) DEC FL0At(6) STATIC ff)lT(O)* 

ROWMIN(IOO) ' FIXED BIsj(31),/» LOC :L0WEST^ST IN ROW 
ROWBASEilOO) FIXED BIm(31)I>» ITEMS 'B£Fi)RE ROMiINJ)D 

/• 

IF INDEX(RTP»«WRITE«)>0- THEN wRITESW=«l«Bl 

ELSE WRITESW=«Q»BI ^/ " 

IF INDEX(RTP* •N0SIN6»)>0 THEN SING=«0«B; . 

ELSE SING='1«BI 

/♦ NEAREST NEIGHBOR • 
GAMHAsO; ' • ■ ' 

/* OTHER SETUP- ^ 



•/ 

*/ 
*/ 

•/. 



00001 
.000Q2, 
000(13 
00004 

odoosr: 
0000.6; 

00007 
00008 

po.ooi?.. 

00010 

oooii-- 
„o.oo i2 : 
oooil ; 
ooouy' 

00015 : 

00016 i 
00017'. 

J)QD18 ' 

00019 ; 

00020 

o.o.a2i ' 

00022 
00023' 

00025 
00025 ' 
MO.2.7.1 
00028 
000^9 ; 

m(i3.a.= 

00031h 
00032 ; 
00033: \ 
0003<:t. 
0.0035 ; 
I'M 03)6. 
' 00037:_ 
S 0.0038 
.J).0fl.3.9 
OOO^O . 
OOO^l ■ 



J 



ERIC 
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"4 * ^ 



■DM07P002 



01/26/77 PAGE .2 



/ 



•OSTMIN=OI 
. DO 1=1 TO Kfo) 
CURR(I>=U 

DOCTRM=0 I 
DOCOOC=O.I- 
ROWMIN=OI 
0OCDOC(1&)=2l 

GROUP,ASSOC=:OI 
INERR=0; 
- ON UNDERFLOW BEGINt 

PUT PAGE 0ATA(ECI,EC2,CUR»LAST,DIST»I0»LNEC1) I 
^ PUT SKiP(2) OATA(TOP»DOCMAX) ; 
RU.T_SKiaiai_ EaiL(LN-CflL»NXI(0) ) lEXSi tB (5) )>} 

GOTO OUMPPHI 

end; ^ ■ . 

- ■ ON ERROR BEG IN I 

INERR=INERR*l; IF INERR>1 THEM GOTO EOP; 
■ ^ CALL DUMP I 
^ - - GOTO DUMPPH? 
' ENDI 

ON ENDFILE(SYSIN) GOTO STAGE2j 



RD: 



/* 

/• READ DOC-TERM ARRAY 



»/ 



_ GET EpiT{D0CN0,D0CCNT»TCNT) (F^3) »F(3) »F(3) ) I 
PUT SKIP E0IT{D0CNO#DOCCNT»TCNT) (F(3» ,X(2) ) ? 

IF OOCNO=0 TJiEN.-SIGNAL ENDFIL.c (SYSIN) t 

SI.ZE(DOCNO)=DOCCNT| 
DO 1=1 TO tcnt; 

-GET EDmTRMNO) {F(33)} 
PUT EDIT(TRMN0)(X<2)»F<3))J 

IF TRMNO=0 THEN L)OCTRM(DnCNO,0)=DOCTRM(DOCNO»0) ♦U 

ELSE. .pOCTRM(DOCNO»TRMNO)=i f 

IF tr'mnotrmmax then- TRMmAX^TRMNGI 

ENDI 

IF DOCNODOCMAX then UOCMAX=DnCNO| ' ' 

GET SKIP? 
GOTO RDJ 



STAGE2: 



TOP=0OCMAX? 
MULT1=2*D0C^AX| 
DSTMIN=2l 
DO 1=1 TO DOCMAXi 
MULT2=I-lt 

.BASI=MbLI2»(MULTl-n/2l - 
ROWBASE(I)=BASH 

TuE 



/« DOCTRM COMPLETE HERE I NOW DO DOCDOC 
/• 

/♦ • ' 

/•» REMOVE REDUNDANT ROWS BY MERGINGf - 
/*» AND COmPRESS DOCTERM TO FILL iHOLES 



00042i 

000431 

000,4^ 

00045:' 

00046, 

005481 
. -0^049' 
-0X)056: 
OOOSlj 
t 00052: 
..QOQSa.. 
00054 
00055; 
--QDHSfir 
0005~t; 
60058.{ 
J.-OOOSSi? 
: 000601 
! 0006T 

; Jio.062;^ 

i 00063: 
00064 i 
.'^0065.' 
00066 
00067 
JlOSihA. 
I 00069 
: 00070 
:^„000li.: 
00072- 
00073 
-00034. 
i 00075 
: 00076' 

•♦--ojoojjl; 

0007^ 
0007.9 

-OflDJBii. 

■ oooat; 

00082 

. _o.o.o.ai 



/♦ LEGAL MAX IS..1 



ERIC 



DOC-DOC MATRIX IST STORED IN 



C33158 



•/ 


00064 


•/ 


00085 






♦/ 


00087 


»/ 


00088 


•/ ao.ofl9L 




;ooo9o 




00091 


t/ . 






00093- 




00094 








00096 


»/ 


00097 



01/26/7X PAGE. ■ 3 



b.MOTOOOa 



/» REDUCED FORM. ONLY THOSE ITEMS 
/♦ Dn{I»J) WHERE J>I ARE KEPT 

/•-Item dd(i?j^is in DocDor.(LOC) 
/• Where 



/• 

/» 



l.OC= 



(I-l) (2N-I> 



J - X 



/• 'OO A ROW OF DO 



»/ 
♦/ 
♦/ 

♦/ 
♦/ 

*/ 
•/ 



do j=i+1 to docmaxi 
un»int=o; 

IF SING then UN=DOCTRM(I,0)*DoCTRM(J<0) ; 
DO t=1- to TRMMAXI 

IF D0CTRM(I,T) >0 THEN" 

IF doctrm'J»t)>o then 

JN J= I NT ♦DOCTRM < I t r ) ♦DOCTRM { J t T ) ; 

/• INT IS intersection SET- SUM*/ 

un=un*ooctrm(i ,t) ♦doctrm{ j,t) i 
^ ■ - /• un is union. set sum •/ 

^ end; 

• -LOCsJ-I^BASH 
' lNT^^INT/2;' ; 
XI=INT; . \ 

UN=UN-INT; •/♦ DON'T COUNT TWICE */ 

XU-?UNJ 
IF UN=0 THEN XU=l; 

DIST=1-{XI/aU) I 
DOiDOC(L.OC)=UIST; 
IF'DIST<OOCDOC{ROWMtN(I) ) 
IF DIST<0ST-^IN THEN DO; " 
0STM1N=UISTI 
Hl = n ' HJ=J: ^ 

end; 



THEN ROWMIN(I)=l,OCl- 



/• SAVE CLOSEST PAIR 



•/ 



end; 



end; 



/• 



•/ 



PUT PAGE! 

IF WRITESW THEN 00; 

DO 1=1 TO TRMMAX; ' • ' 

00 j=I*l TO TRMMAX ; ' • " ' 

. DrsTsuM=o; numdist=o; 

DO 11=1 TO OOCMAX; 

•'.if D0CTRM(I1,I)=1. THEN 00; 

DO Jl=l TO docmax; , ' • 

' IF D0CTRMU1,J)=1 then DO; 

numdist=numdist*i J 

IF J1>I1 then DISTSUM=DISTSUM*DOCDOC{RbwBASE{Il)*Jl-Il) ; 
ELSE IF I1>J1 THEN DISTSUM=DIStSUM*D0C00C (ROWBASE { Jl) ♦U-Jl ) ; 

End; 
end; 

end; • . -. 

end; 

TEiRMl = I; TERM2=j; TTOIST=DISTSUM/NUMblST ; 
WRITE FIUE{0UT> FROM (DISTnOOE) ; 
END! . . - - - 

END I «' 



00098.' 
0009.9 ^ 
00100 
OOlOi - 

00102.: 

00103 

0,0104 ; 

00105* 
00iO6 • 

0010,7;, 

00108 : 
.0010.9 
00110 
00 111 

.00.112 
001 1-3: ; 

00114: 

jO.0115^ 
00116 
00117 : 

o.ai ijB. 

001*19 
00120 
00.1^1 
00122 
00123 
00.124/ 
00125 
00126, 
00127. 
00128 
00129 
00130 
00131. 
00132 
00133. 
00134 
00135 : 

00137 
/O0138. 
Ml 3.9. • 
00140, 
0014L 
-Oaiit^ 
00143. 
0Cl44 
Q.0.145. 
00146 
00147: 
-OXilAa 
00149 
00150. 

Mi5i 
00152 
00153 
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ENDt 



/* DOCDOC 



IIMAX=2«T0PI 
ALLOCATE. ELTSI 
ELTSsOI. -- 
STAGE3: 

TOP=TOP*H 



complete; 
' /• 



NOW MAKE CLUSTERS 



V 



•/ 

-•/ 



GROUP ( CURRXHU.) .» GROUP.tCURR-(-H J )_)-=TOP I 
ASSOC {CURft (MI )) ♦ASSOC (CURR(HJ) )=DSTMINI 



/* SET NEW GKOUP NUMBER 



♦/ 



PUT 



'fTOPf • 



FROM 'rfCURRCHD-,' 



IF 



THIS DELETES ROW HJ 
RE-USE ROW FOR NEW GROUP 



SKIP EDIT (♦FORMED 
• At-D-ISTilNCE:-i»DSTMIN) 

. (AfF(3) f AfF(3) fA,F(3) ,A»F(7,5) ) ; 
SIZE(CURR(HI) )>=SIZE(CURR(HJ) ) THEN DO* 
- - .ELTS.(T-OR-»aJ-=CURR-tilI).r _ . 
ELTS(T0Pf2)=CURR(HJ) J 

END I 

ELSE oaj 

ELTS(TOPfl)sCURR(HJ) J 
' ELTS(T0^2)=rURR(HI)| 

- - ENaj ^ I 

SIZE(T0P)*SIZE(CURR(HI) )*SIZE(CURR(HJ) ) J 
ALPHJ=SIZE(CURR(HI) )/SiZE(TOP) I 
. ALPHl!CsSIZ£.lCURR.(HJJ )/SlZE(TOP) I 
, CURR(HJ)=0» . ' /• 

CURR(HI)^T0PI /• 
BASX=ROWBASEL(MI) I 
8ASK=R0WBASE(HJ) ? 
DOCDOC (BASX*HJ-HI ) £2 1 

/• 

/« NOW FIiL NEW DISTS IN ROW X 

^ 0JK=DSTMIN| 

O DaTMiN=2r 

/■ R WMIN(HI)=:0I 

DO i = l TO DOCMAXi. 

IF CURR(U=0 THEN 
IF HI=I THEN GOTO 
BASI=ROWBASE(I) t 
DIST=2I. 

IF KHI THEN DO! 

L0C2=BASI*HI-II 

DU=D0CD0C{L0C2) I 

IF L0C2=R0WMIN(I) 
ENDI- « 
ELSE DIJ=DQcpOC{bASXtI-HT) I 
IF I<HJ" TftEN "DO I ♦ ... 

L0C2s;8ASI*HJ-II 

.OIK=DOca0C(LQC2) I 

DOCDOC (L0C2) =21 

IF LOC2=ROWMIN(I)^ T^EN 
ENDI , . 
ELSE 



»CURR(HJ) 



GOTO Sk 
SKIPI 



SET NEW DISTANCES TO ROW X 
PI /•A DELETED- R"QW' 

/• SKIP ROW X ITSELF 



TwEN 




•/ 
•/ 



♦/ 
♦/ 



•/ 

•/ 

•/ 



0IST=-1I 




DIK=0OCDOC(bASK*I-H >) I 

DIX=ALPHJ«DlJ*-ALPHK«DIK*>5ETA*DJK*GAMMA»ABj^^DIJ-DIK) I 
IF I<H1 JHEN DOI 

LOCsBASI*HI-II 

D0C0QC(L0C)=DIX| 



ERIC 



C3 



160' 



•bM0700D2* 



PAGE 



SKIP: 



ERIC 



IF DIX<DOCDOC(ROWMIn(I) ) THEN KarfMlN ( I ) =LOC I 

end; 

ELSE 00 » . 

. LocsBAs;c*i— <i; / 

•06CD0C(L0C)=DIX; / 
,IF DIX<DOCDOC(ROWMI.m(HI) ) THtN RoiMIN(HI) =LOC; 
' ENDI . 

IF cisKO Then duj . 

L0C3=BaSI*Dv)CMAX-II 
DIST=2| 

DQ L0C2=BASI*1 TO LnC3; 

IF D0CU0C{L0C2) <D1ST THEN-O'o; 
DrST=DOCDo"< (L0C2) I / 

Rowmin('i)-~l0C2; 

END! / 
ENDI / 

end;', ' 

IF DOCDOCCROtfMlNU) XDSTvIN THEN 00} 
HHIrn ■ • - 

HHJsROwMIN(I,)-BASI*Tl 
OSTM-IN-OOCOQC{ROWM.I m(I) ) ; " 
ENDI ' 

o 

END} 

IF DOCOOC{ROWMIN(HI) ) <DSTMIN .HEN 001 
• .?'*HHI=HII . ,• » 

, HHJ=R0WMIN(HI)-BASX*H-I| 
•DSTMIN=DOCOOC(ROaMIN(HI).) I 

ENDI 
HI=HHI} 
HJsHHJ-f 

IF 0STMIN<2 THEN GOTO bTAGE3l 

/♦ • ♦/ 

ALLOCATE ^iXTI , • 

•'form=» *f- 
LN=o; 

NXT(TOP)-0| ^ 
CURtFIRSTsTOPI 

LAST=OI , ^ . ^ / 

. 00 rfHILE(CUR^=0) t 

-- IF CUR>DOCMAX TH6N Odl 

ECl»NXT(LAST)=ELTS(.-URtl) I 
EC2«NXT(EC1)=ELTS(C.:R».2) I 
^NXT<EC2)=NXT(CUR) r 
'DlST={ASSOC(fc.Cl)«10 •)*.5l 

• . id=oist; 

formo (ec! ) »forho (ec? ) =formo (cur) i 
if ln(cur)-'=0 then ,01 

LNECl»LN(ECl)=i N(CUR) I 
IF F0RM(NXT(ECr5)tLNECl)-'=»*» 

Then F0RM,EC2fkW5Cl-)=» *i 
ENDI o ^ r 

ELSE LN(EC1)=I0} 
IF ID>0 THE-M 

'FORMTECltIO) tF0RM(E-2tID) = »»» r ' 
LN(fC2)=I0l 

03^61 



•J 



ooai'o 
ooni 

00212; 

0021-1 

0021^ 

00E15. 

00216. 

00217'' 

00218- 

00219 

00220. 

00221 

00222 

00223. 

00 224.: 

00225 

do 226 

00227 

0022^: 

• 0022^ 
00230 
O023'i 
00232- 
00233 
00234- 
00235 • 
00236 
00237 
00238 
0023.9 : 
06240 

00241 ; 

.0.02.42: 

00243 

0024^ 

0.0246 . 

00246 ; 

00247 

00.246 

00249' 

0025() 

00252 

00253 ■■ 

0.Q25-4 

00255 

00256 

.a0252. 

00258 

00259 

.0.02610- 
0026.1 
00262 

_ -0J)26.3 
00264 

00265 ■ 



CUR=ECU 

end; 

£LSE_X)04 

LAST=CURI 
CUR=NX.T(CUR) I 



END} 



-00-1=0- -Ta4)0CMAX4. - 

DO J=l TO LN(I) I 

F.ORMd, J) = «*» t 
-END! . 
END I' , 
PUT PAGEt ■ 

PUTlEOItU- • ')UUII)j.| 

DQ Q=.l TO 1 BY ,11 
■ . PUT EDITfQ) {X{7) »F(3,1)).:,' 

I=NXT{0)'| ' 
00 WHILEd-irO) I 

•l?in._StaP EDLTa«FORMO(I)< 
I^NXTd)! 

END; 

. - " /* • •/ 




A) tXte) lAQOO) ) I 



V ' . /« WE SHO iLD NOW BE#ALL DONE 

- ' . i • /» 

oumpph: • ■ 

PUT page; 

PUT StaR\DAlAXDOCMAX,T.pMMAX) K 
PUT SKIP(3) DATA(OSTMIN,HltHJ,HHI,HHJ) J ' 
J=( {00CMAX*D0CMAX)-00CMAX)/2? 
-PUT SlClP{ai_OATJLJvUJ_ 

IF OOCMAX=0 THEN DOl . ' . ^ 

PUT SKIP! 

DO L=l.jrQ. J} . , 

PUT EOIT<I,»:t,COCD0C{I)) (X{3)»F(4), { 1) vF (TtSHl 

end; , 

DUMPJ. .RROC, RE^QSOERI 

PUT PAGE I . r 

DO 1 = 1 TO DOCMAX; . . • ' - 

PUT SKIP £011 tj ♦RCWBASE H) HF O) If 15) ) t i 
PUT EOIT(CURR{I))(X(3)»F{4).5 

PUT EDIT{SIZE{CURR{I))) {x(2)»F{3))f- 

PUT EDIT(RQWMlN.{U) {X(2),Fn)); 

PUT SKIP! 
00 J=I*1 TO DOCMAX I 

PUT EDIT ( J, • ; • .DDCDOCCROWBASE ( I) ♦J-I) ).(X (3J ».FT^)-, A«F tZ»S) 
ENDJ ' 
ENO» . . 

END DUMP J / 

END I > ' - 

PUT SKIPI3) ; * . * «fe 

00 1=1 TO TOPJ _ . , 
PUT SKIP, EOIT{I»GROUP(I},ASSOC(I)) (F ( O tF (5) ,X (3) »F { 7,5) ) ; 
PUT EDlT{ELTS{I,i),ELTS(I,«;),LN{I)i ((3) {X(3),F(5)))I 

PUT EDIT(NXT{I)){X{3),f'(5))? 
ENDl , 

EQRJL ' . _ _ . . _ . 



) t 



ERIC 



CLUSTER! 



C37162 



i 

00266 
00267 
-MZ6B 
002^9 
0027d 

00272 
00273 

00275 
00276 

00278 
002T9 
00280. 
00281 
00282: 

.im2al 

0028^ 
00285; 

. .oo^a6- 

Q0287: 
00288 
.-01)289- 
00296: 
00291 
-.Qi)292- 
00293 
00254 
-00295. 
00296 
00297. 
. 0029.8, 
00299^ 
00300 
. „a03j(U- 
003.02 
003(^3 
„ ,.00304. 
. •"^00305 
00306 
,00307 
00308 
00309 

,0i)3m 

00311 
00312 
_-0il3i3. 

00315 

003 1.6 

003^7 
00318 
-_ai)aiSL 
00320 
00321 

00322 
00323; 

_Lli)03at 

00325 



« 
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APPENDIX D 



C ■ > 



J 



er|c , . - . ' Dii6:( 



^N0 eiO^VST 



to 



CAbtC" 



CAsec 



^0 



66 



?3 



^3 



A-sATtMtNT 
AaOu .^JtvAL 



60 



100 
100 
7d 



0 
0 
13 



'•rJtUi AN " 
AotKHttFlO^ib 
_Aoi L 1 1 V 



100 
100 

50 



0 
0 
5A 



0 
0 

_<^A 

0 
0 
9 



urtLATlUN 
, A-3LC [ 



n 



100 



77 

:>6 



Ao'<Ao 1U\^ 
Ao^WOAm 



AcJb 

At^SLlblC 



"77' 

,i 
6 



100 
33 
100 



0~ 

' 0 

0 
70 

0 



ln 



0 
io 
0 



100 
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